Title: SpatialLM: Training Large Language Models for Structured Indoor Modeling
Venue: NeurIPS 2025
Accepted: yes
Keywords: Structured Indoor Modeling

Large Language Models

Spatial Understanding

3D Reconstruction
Abstract: SpatialLM is a large language model designed to process 3D point cloud data and generate structured 3D scene understanding outputs. These outputs include architectural elements like walls, doors, windows, and oriented object boxes with their semantic categories. Unlike previous methods which exploit task-specific network designs, our model adheres to the standard multimodal LLM architecture and is fine-tuned directly from open-source LLMs.

To train SpatialLM, we collect a large-scale, high-quality synthetic dataset consisting of the point clouds of 12,328 indoor scenes (54,778 rooms) with ground-truth 3D annotations, and conduct a careful study on various modeling and training decisions. On public benchmarks, our model gives state-of-the-art performance in layout estimation and competitive results in 3D object detection. With that, we show a feasible path for enhancing the spatial understanding capabilities of modern LLMs for applications in augmented reality, embodied robotics, and more.


Peer reviews:
comments: SUMMARY:
This paper introduces SpatialLM, a 3D point cloud multimodal large language model for structured indoor modeling. SpatialLM regards 3D modeling as a scripting task, thereby leveraging the outstanding programming capabilities of large language models. The authors created a large synthetic 3D dataset, the SpatialLM dataset, which surpasses existing datasets in both quantity and scale. They conducted extensive experiments to demonstrate the effectiveness and generalizability of their model.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The introduced synthetic dataset of 12,328 indoor scenes is a substantial contribution, enabling extensive training and benchmarking.
2. The author conducted extensive ablation experiments on the design of the point encoder, the selection of spatial resolution, and the training schedule.

Weaknesses:
1. The motivation and approach of SpatialLM are similar to SceneScript, raising concerns about incremental novelty.
2. The improvements of SpatialLM over SceneScript in the layout estimation task are modest (+1.3%/+0.8% at IoU@0.25/0.5). Additionally, SpatialLM does not surpass V-DETR, which does not use LLMs, in 3D object detection tasks.
3. There are some structural issues in writing. The related work section could be presented earlier.

QUESTIONS:
1. What are the advantages of scripts over box texts (like \box{x, y, z, w, h, l} or [x, y, z, w, h, l])?
2. What is the incremental novelty of SpatialLM over SceneScript?  I believe that the format of the output script is not a problem for large language models.
3. What are your rules for selecting scenes/categories? Are there any special considerations to ensure diversity and avoid a long-tail distribution?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 1
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a LLM tailored for structured understanding. SpatialLM predicts architectural elements (walls, doors, windows), and oriented object boxes with semantic categories from 3D point clouds. It follows a standard “Encoder-MLP-LLM” architecture and is jointly trained for each downstream task. SpatialLM is evaluated on two downstream tasks: (1) layout estimation on Structured3D dataset, where SpatialLM outperforms state-of-the-art models RoomFormer and SceneScript, and (2) 3D object detection on ScanNet, where SpatialLM outperforms SceneScript and is competitive with V-DETR.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The paper presents a large-scale (*12,328 scenes*) synthetic SpatialLM dataset with layout and object annotations. The visualization shows that this dataset is of high quality in terms of realism.\n- The paper presents a unified model that can accept language prompts to solve different 3D structured understanding tasks. The architecture follows standard multimodal LLM and can leverage the rich-knowledge encoded from pretrained LLMs.\n- There are some interesting ablations and empirical study on point cloud encoders and training schedules.\n- Strong performance: SpatialLM achieves state-of-the-art performance on layout estimation of 3D scenes on Structured3D dataset and competitive performance on 3D object detection on ScanNet.\n- The paper is clearly written and very easy to follow. There are also many nice visualizations, making the results more convincing.\n\n**Weaknesses:**\n\n- Even though SpatialLM uses the same architecture for different tasks, it is not universal:\n    - It requires dedicated training for different tasks and datasets.\n    - And the training involves training all the three components of the “Encoder-MLP-LLM” architecture for best performance (tab 4), which defeats the purpose of large and universal models.\n- This paper aims for large language models and one important consideration should be model complexity and training data during evaluation. For tab. 2, 5, 6, please add a column of number of parameters so that readers know how model complexity affects results. For tab. 5, 6, please also add a column of training data to make the data factor more clear.\n- Some statement is not precise or wrong:\n    - L24-26: the paper states that the first contribution is that “*regard the structured descriptions as scripts of a general-purpose language (i.e., Python) and propose to predict the language in text form*”. However, I think this is what SceneScript already does. What’s the difference between this contribution from SceneScript?\n    - L297: “*RoomFormer [58] proposes to train a two-stage Transformer-based network to predict corners and rooms.*” I think RoomFormer is a single-stage network using two-level queries, not two-stage.\n- Organization of the paper: I feel the naming is misleading. To me, Section 2 “An Empirical Study on Point Cloud Feature Alignment” mainly discusses SpatialLM, while section 3 “SpatialLM” is actually the experiment section. Therefore, I suggest to rename section 2 as “SpatialLM” and section 3 as “Experiments”, or something indicative.\n\nQUESTIONS:\n- What’s the difference between the claimed 1st contribution from SceneScript?\n- Structured3D is multi-room dataset while ScanNet is single-room dataset. Does SpatialLM do inference on multi-room directly or can only do inference in single-room manner?\n- Section 2.2 explored different point cloud encoders. However, eventually the point cloud encoder still needs to be trained jointly for each downstream task (tab. 4). Also L198-199 says “*current pre-trained point cloud encoders are not as versatile in supporting downstream tasks yet.*” This motivates the question: is pretrained point cloud encoder really necessary? How about just initializing the encoder (e.g. SPTv3) with random weights and then jointly trained with MLP and LLM?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a framework for 3D visual grounding using pretrained language models augmented with spatial inductive biases. The model takes as input a structured scene layout consisting of object labels and their associated 3D attributes (position, size, orientation), and predicts the 3D bounding box corresponding to a natural language query. Spatial reasoning capabilities are introduced into a transformer-based language model by adding spatial embeddings and modifying the attention mechanism to reflect geometric proximity. Experiments on synthetic and real-world layouts demonstrate improved grounding accuracy over baseline models that lack spatial awareness. The approach is positioned at the intersection of 3D vision and multimodal language reasoning, with contributions focused on task-specific integration rather than architectural or theoretical innovation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper addresses a well-scoped problem at the intersection of 3D visual grounding and language-based spatial reasoning. It presents a clean architectural adaptation of transformer-based language models by introducing spatial inductive bias through position-aware embeddings and attention masking strategies. The proposed formulation is modular and lightweight, requiring no changes to the core transformer mechanism, which enhances reproducibility and compatibility with existing pretrained language models.\nEmpirical evaluations span both synthetic environments and real-world datasets (ScanNet), with quantitative metrics showing consistent improvements over non-spatial baselines. The experimental design includes comparative analysis with spatial relation prediction and 3D bounding box localization, both of which validate the proposed enhancements. The use of object-centric representations as input provides computational efficiency and avoids the need for raw perceptual processing, making the system well-suited for layout-level inference tasks. Furthermore, the dataset construction, training setup, and evaluation protocol are clearly described, improving transparency and replicability.\nConceptually, the paper explores how structural priors can be aligned with the attention mechanism in LLMs to support grounded reasoning. This contributes to ongoing discourse in multimodal learning on how domain-specific biases can be embedded into general-purpose models.\n\nWeaknesses:\nDespite practical performance gains, the architectural contributions are modest. The proposed spatial embedding and attention mask mechanisms mirror existing practices in relational modeling and transformer locality bias, offering no new theoretical or algorithmic innovations. The model’s dependence on clean, discrete scene layouts means it cannot operate in realistic settings where perception is partial, noisy, or continuous.\nThe spatial bias mechanism is treated as a black-box utility without detailed analysis of its effect on attention dynamics, training stability, or generalization. There is no investigation into whether the model internalizes spatial hierarchies or relational patterns, nor whether the improvements stem from inductive bias or training data regularity.\nMoreover, the model is evaluated only in bounded task settings involving synthetic or preprocessed layouts, and its performance under distributional shift, ambiguous language input, or sparse spatial cues is not tested. These omissions weaken claims of general-purpose spatial reasoning.\nFrom a broader research positioning, the work contributes primarily to task-specific improvements within the domain of 3D vision and grounding. It does not generalize to open-vocabulary grounding, sensor-based reasoning, or joint perception-language planning. As such, the contribution is best viewed as an applied extension of language modeling for layout-conditioned reasoning, rather than a conceptual advance in foundational representation learning.\n\nQUESTIONS:\nHow sensitive is the model’s performance to the accuracy or sparsity of the input 3D layout? Would the method remain effective under noisy or incomplete object representations, as would be expected in real-world perception pipelines? How does the spatial inductive bias affect attention behavior or sample efficiency during training? Is there evidence that the proposed modifications improve generalization to unseen spatial configurations or object arrangements beyond the training distribution? Finally, could the same spatial reasoning be achieved by finetuning with task-specific data alone, without architectural modifications?\n\nLIMITATIONS:\nThe method assumes access to structured scene layouts composed of object-centric tokens with precise 3D attributes including class, position, orientation, and size. These layouts are either synthetically generated or derived from processed ScanNet scenes, relying on accurate object detection, segmentation, and pose estimation pipelines. The performance and applicability of the model in scenarios where such structured representations are incomplete, noisy, or unavailable are not addressed. While the proposed spatial inductive bias improves performance on layout-conditioned reasoning tasks, it is unclear how the model would behave when exposed to real-world uncertainty, such as missing objects, ambiguous queries, or spatial occlusions.\nAdditionally, the approach is strictly conditioned on discrete object instances, limiting extensibility to raw sensory data such as point clouds or voxelized scenes. The spatial inductive bias is implemented via embedding injections and localized attention masking, but its impact on model interpretability or reasoning trajectories is not analyzed. There is no investigation into whether the model develops spatial priors or relational structures beyond what is encoded in the training data. Generalization to unseen object configurations, novel spatial relationships, or out-of-distribution compositions is not systematically evaluated, which constrains the scope of the method’s robustness claims.\nLastly, the model does not perform end-to-end reasoning from language to perception but instead operates in a middle-layer abstraction, making it difficult to assess its utility in embodied tasks or real-world robotic applications without substantial integration with upstream vision modules. More broadly, the research is situated within a structured-layout paradigm where perception is abstracted away, which limits its relevance to fields focused on holistic scene understanding or sensor-to-action pipelines. This constrains the potential impact of the work beyond its immediate 3D layout reasoning setting.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SpatialLM, a VLM finetuned for structured indoor modeling tasks. The authors curated a large-scale 3D dataset in simulation and finetuned a small language model to take input from point cloud features and output Python-style text language to annotate objects and room layouts in 3D. Experiments demonstrate the feasibility of this fine-tuning paradigm and the effectiveness of SpatialLM compared with baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. Methodology. This paper proves the feasibility of directly finetuning an LM, even a small one, for generating 3D indoor modeling, which is a meaningful augmentation to language models, especially useful for real-world applications.\n2. Dataset. The curated dataset seems to be of high quality and produces good pretraining effects (Tables 5 and 6). Details of the dataset are also presented in the appendix. If the dataset can be open-source as promised by the authors, it will be of great use to the community. Can the authors confirm the open-sourcing plan?\n3. Solid performance. Design choices are validated with experiments in Section 2. SpatialLM produces strong performance on layout estimation and 3D object detection tasks. Further finetuning on task data gives even better performance, which is a good sign for the potential usefulness of the model.\n\n**Weakness**\n1. The necessity of using a language model for the tasks. As the paper states, the model is currently confined to a predefined set of objects, which raises the question of the need for a language model. It is also a waste of MLLMs' generalizability. For such a fixed set, can a strong 3D point cloud feature + a strong segmentation model suffice for the task? Meanwhile, the paper justifies the advantage of using the Python language as the output by stating its editable and easily extendable. However, such features are not demonstrated in the paper. It would be nice to see if prompting the language model can help edit the layouts.\n2. Robustness. It seems the performance is mainly tested assuming perfect 3D inputs. Only Section 3.3, the model is tested with reconstructed inputs. Although it does show some robustness, the experiment is only qualitative. Is it possible to roll out quantitative experiments on datasets that have 3D groundtruth but use reconsrtructed 3D as input? What would the performance be like?\n\nQUESTIONS:\n1. Line 167: What exactly is this 5-level structure?\n2. Can the model be potentially extended to handle dynamic scenes when objects and layout change? If so, what is needed?\n3. How would you augment the object sets for different application scenarios? Does that require a significant amount of additional training?\n4. After training, does SpatialLM still possess general language generation capabilities? Can you freely prompt the model to answer scene-related questions?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}