Title: Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D indoor scene synthesis

text-to-3D layout generation

spatial reasoning
Abstract: Realistic 3D indoor scene synthesis is vital for embodied AI and digital content creation. It can be naturally divided into two subtasks: object generation and layout generation. While recent generative models have significantly advanced object-level quality and controllability, layout generation remains challenging due to limited datasets. Existing methods either overfit to these datasets or rely on predefined constraints to optimize numerical layout that sacrifice flexibility. As a result, they fail to generate scenes that are both open-vocabulary and aligned with fine-grained user instructions. We introduce DirectLayout, a framework that directly generates numerical 3D layouts from text descriptions using generalizable spatial reasoning of large language models (LLMs). DirectLayout decomposes the generation into three stages: producing a Bird's-Eye View (BEV) layout, lifting it into 3D space, and refining object placements. To enable explicit spatial reasoning and help the model grasp basic principles of object placement, we employ Chain-of-Thought (CoT) Activation based on the 3D-Front dataset. Additionally, we design CoT-Grounded Generative Layout Reward to enhance generalization and spatial planning. During inference, DirectLayout addresses asset-layout mismatches via Iterative Asset-Layout Alignment through in-context learning. Extensive experiments demonstrate that DirectLayout achieves impressive semantic consistency, generalization and physical plausibility.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed a model that can generate renderable indoor scenes from text prompts. This is achieved by training several specialized LLM agents. The authors decompose the task into BEV layout generation, 3D lifting, and refinement, each stage handled with a specialized agent. The agents are trained with guidance from other VLMs/LLMs such as GPT4o through SFT and DPO. The authors show visual examples of the generated scenes and do and numerical comparisons with a few baselines. I find the overall quality of the generated scene pretty good and plausible. Overall, this is a technically sound and well-written paper solving a useful task, but I have some questions regarding evaluations and limitations.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n- Decomposing the task into layout generation, 3D lifting, and refinement makes sense intuitively.\n- Training specialized agents with SFT followed by RL-based finetuning, although being a standard pipeline for LLM finetuning, makes sense in this task.\n- The paper is well-written and easy to follow.\n\nWeaknesses:\n- Evaluation: The authors do automatic evaluations with VLM in Tab 1, which is valuable, but the authors should also add human preference evaluations, which is more straightforward and convincing. \n- Baselines: The authors compare against a few baselines using LLM agents, but it would also be interesting to see how rule-based methods work. For example, the method used in Infinigen-Indoors can handle pretty complex layout planning. Since there aren't many objects in the rooms generated, my guess is that the rule-based method would be a pretty strong baseline.\n- There are some other limitations I find that I want to discuss, but I put them in the Questions section.\n\nQUESTIONS:\n- How is the shape and size of the room determined? Is it a fixed size across training and validation, or input to the layout generation framework? In the real world, rooms are often not square-shaped and may have additional walls inside them. Is your model capable of handling these additional constraints?\n- The scenes shown are kind of toyish and empty, with only a few objects. How well can your model scale up to more complex scenes? For example, an auditorium with hundreds of chairs, or some special types of rooms such as a conservatory in a botanical garden filled with plants? The limitations section touches on this a little bit, but I think the discussion is not enough. What is the exact boundary of complexity that the current model can push? Can we scale it up in the future with more data/more GPUs, or are there any fundamental issues you would foresee?\n- I'm very interested in seeing how well the proposed method can handle the composition of objects (for example, many small objects placed on a shelf/table). In Fig 3 last row, are the objects on the table generated separately and placed on the table, or is the \"table with object\" a single asset? The bathtub with ducks example is interesting, but in general, I would like to see more examples of how these compositions are achieved and gradually modified to be physically plausible.\n\nLIMITATIONS:\nThe authors discussed limitations in inference time and complexity constraints, but I want to see more in-depth discussions on the complexity constraints (see Questions). Also I would like to see some failure cases.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes DirectLayout, a framework that generates 3D indoor layouts directly from textual descriptions. Unlike existing methods that either regress numerical layouts from data (with poor generalization) or use intermediate symbolic representations (with rigidity and poor fine-grained control), DirectLayout introduces a three-stage pipeline:\n\n1. BEV layout generation, enhanced with Chain-of-Thought (CoT) reasoning.\n2. 3D lifting using commonsense priors.\n3. Iterative asset-layout alignment using feedback from dual evaluators (LLM + VLM).\n\nSTRENGTHS AND WEAKNESSES:\nThe paper clearly defines the limitations of existing paradigms and motivates its simple yet effective algorithm based on Chain-of-Thought (CoT) reasoning and numeric generation. The model outperforms LayoutGPT, Holodeck, and I-Design across nearly all metrics.\n\nThe main weakness of the paper is the lack of evaluation on out-of-distribution generalization to novel categories of scenes not present in the training set. In particular, the method relies on heavy fine-tuning through Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), which may hinder its generalization compared to methods that keep the base LLM frozen.\n\nAnother limitation is that, being based on a Bird’s-Eye View (BEV) representation, the method is not well-suited for handling scene layouts with significant vertical stacking or clutter—such as books on bookshelves. While this is not the primary focus of the paper, it does limit its broader applicability.\n\nFinally, although the authors use LLM-based evaluators (as in LayoutVLM), they do not report any human studies to directly assess alignment or plausibility.\n\nQUESTIONS:\nHow sensitive is the performance of your BEV Layout Generator to the quality of the GPT-4o-generated CoT traces? Did you experiment with noisy or incorrect CoT traces to evaluate robustness?\nYour CoT-Grounded Layout Reward combines both VLM and LLM feedback. How critical is this dual-evaluator setup?\nHow well does DirectLayout generalize to unusual room types or unseen object categories not represented in the 3D-Front dataset or training prompts?\nWhat are the most common failure modes you’ve observed?\n\nLIMITATIONS:\nYes. But there are more limitations, especially on the usage of BEV as the intermediate representation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies 3D layout generation. The proposed pipeline follows the widely used LLM-agent pipeline, trained SFT using GPT-generated CoT data, and then standard DPO. The contributions of this paper mainly lie in the engineering tricks within this pipeline, including the specific reward designed for the alignment between CoT and the generated layout and the final interactive asset-layout alignment for refinement. Both qualitative and quantitative measurements show that the proposed method is effective.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n- The high-level ideas of the method are straightforward to understand. The engineering tricks are relatively well-motivated.\n- All the visualizations in this paper are impressive, showing that the generated layout is reasonable (while also thanks to the out-of-the-box object generator).\n- Both quantitative and qualitative scores show that the proposed method is effective and outperforms state-of-the-art.\n- Specifically, the experiment part considers the use of different LLMs and tests the proposed method in different settings, e.g., open-source only, to make a fair and holistic comparison.\n\n### Weaknesses\n- As the paper employs the widely adopted LLM agent pipeline, with CoT as also a widely adopted extension, even if it might be the first to employ it in layout generation, transferring such a well-studied pipeline to the new task should not be solely regarded as a contribution/novelty.\n- **Crucial.** The method section is not well-written. Based on such a widely used framework, the actual novelties of this paper are engineering tricks. This makes the paper more like a technical report, revealing numerous details; however, on the other hand, these details are still mainly text descriptions with uninformative formulas, which are still far from being reproduced. \n    - For example, Sec. 3.1.2's \"Object-Level Criteria\" introduces the reward function with seven terms. This is very detailed, but okay, as they are related to the main contribution. \n    - However, none of these terms are defined rigidly or explained in detail - simply mentioning \"Measures the reasonableness of spatial relations with related objects\" or \"Checks that spacing between objects supports accessibility and func162 tionality. Overlaps may be acceptable if vertical stacking is plausible\" is far from sufficient, which cannot tell how they are defined and distinguished.\n    - All the formulas (3)-(7) are just uninformative and simple rephrases of texts. For example, (3) is just a simple yet space-consuming explanation of \"the proportion of objects in the layout that satisfy the corresponding requirement\" (L169). All those formulas are just placeholders to satisfy \"there should be a formula\".  \n    - All these make the paper hard to reproduce, even if the engineering tricks are the focus of the paper.\n- Like all LLM agent methods, the inference requires a full round between the LLM planner and the layout generator, which can be time-consuming. Specifically, the proposed method involves an object generator for per-object generation, which further increases the time consumption.\n- As using BEV, the proposed method may not be able to generate complicated z-axis, e.g., a large wardrobe hanging above a table with several fruits on it, under which there is a chair. As shown in the supplementary video \"bathroom_wo_refine\", the model seems to be confused about the height location of the object. This hurts the scope of the proposed method, raising concerns of whether the proposed method is still only generating 2D/pseudo-3D layouts instead of real 3D.\n\nQUESTIONS:\n- I wonder if GPT-4o can generate reasonable CoTs, instead of just generating perfunctory analysis to overfit the answer. It would be beneficial to provide examples of GPT-generated CoTs.\n- The proposed method leverages the visual grounding of a trained model to generate a layout, which reasons on top of the coordinates. I would like to see the raw outputs of the planner, to see how it reasons with the coordinates, and how precise they are.\n- Please provide more explanations about the seven terms in \"Object-Level Criteria\".\n- If the author(s) disagree with the reviewer's claim that \"the novelty of the paper is engineering tricks on top of a widely adopted framework\", please argue the novelties that can be regarded as the novelty compared with the whole LLM agent field to improve general LLM agent designs, instead of the novelties in the layout generation task.\n\nLIMITATIONS:\nThe authors mentioned the limitations. Some others mentioned in \"Weaknesses\" about the lack of 3D z-axis capability.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a new system for open-vocab, languaged-conditioned, indoor layout synthesis. The system uses LLMs/VLMs to (1) iterate on a language specification of a birdseye view floorplan, then (2) convert 2d plans to likely 3d object heights/specs and finally (3) refining object locations. This is comprised of structured prompts for eliciting CoT, describing BEV layouts, and LLM-reward for DPO finetuning. The provided qualitative results are strong, and the method's results are preferred in LLM-based quantitative evaluations.\n\nSTRENGTHS AND WEAKNESSES:\nqualitative results are strong but not plentiful or controlled:\n- strength: the model produces very good layouts with good instruction following, especially in the kitchen and home-theater examples - these feature objects not attached to walls which are minimally constrained and difficult to place correctly.\n- weakness: needs larger & tightly controlled qualitative results\n\t- only one example is shown for each granularity of prompt. we do not know the variance / stability of the method when repeatedly sampled with the same prompt. \n\t- not stated how the prompts were selected - are they representative/random/standardized, or designed to showcase the strength of the authors method?\n\t- ATISS & Infinigen provide large samples of 25+ _randomly selected_ qualitative results in their supplements, this work does not.\n\nquantitative results have two major flaws:\n- Authors follow LayoutGPT in using metrics which are _evaluated by GPT4o_ (with no human study) which I believe is a major flaw in evaluation of this work.\n\t- using GPT-4o as the validation score, when iterating against GPT-4o's criticisms is an integral part of the method, is invalid. This is akin to optimizing against the validation set - if we can see (somewhat stochastic samples of) the final validation metric and change our answer, it is no longer a validation metric.\n\t- I recognize LLMs-as-judges is a widely used paradigm in NLP, but I believe it is invalid for layout as VLMs are not known to possess strong spatial reasoning without significant scaffolding / extra engineering (as evidenced by goals of this work)\n\t- Every baseline method in table 1 provides a user study (Holodeck, I-Design, LayoutGPT). The authors do not perform a user study, they instead cite that LLMs have correlated with users in the past. But this is questionable since the authors method is different & more LLM/VLM reliant than prior methods.\n\t- Out-of-bounds/Collision metrics are quantitative but not sufficiently informative. Objects could topple when settled in simulation, or have their affordances violated (objects blocking chairs/ovens) so these are useful to include but not sufficient as a sole evaluation.\n- Table 1 omits comparison with ATISS, which is well known and which I consider closely related in method. Less critically, the authors could also evaluate DiffuScene. \n\nDiscussion of the method's cost is inadequate. \n- only mention is \"introduce additional inference time\" L315\n- The authors should know the average spend on OpenAI APIs per scene, but the cost is not stated.\n- Tab1 should list costs of each method. Fig.3 could report per-scene costs (is the cost higher for more complex scenes? this would be interesting). See questions below. \n- If the cost is high, is the method better than baselines on an equal-cost basis (baselines run for longer, or authors method is allowed fewer iterations?)\n- The authors fail to argue that the performance-cost tradeoff of adopting their method is worthwhile for downstream tasks. Would roboticist use the method if scalability costs $X/scene in API credits/GPU-time? We cannot know as the cost is not stated.\n- Increased cost would be a predictable downside of their method of replacing constraint solvers with LLM reasoning, which is the core proposal. One gains flexiblity / open-vocab, but loses the computational efficiency of a constraint solver.\n\nDistinction from prior work, especially related to \"constraints\", is imprecise:\n- \"mainly focus on modelling the distribution of indoor scene layouts, wheras our method leverages limited data to learn the underlying placement logic of indoor layouts\" (L85) - what is the difference getwen these two? is \"learning the logic from data\" not an example of learning the distribution?\n- \"explicit logical outputs\" L101 instead of constraints - the methods outputs (L155) are very similar to constraints but are \"solved\" by an LLM\n- \"predefined constraints\"... \"inherently sacrifice flexibility\" L36 - not necessarily - if the predefined DSL is sufficiently low-level (has primitives that are universal to all scenes, not specific to objects), they may apply to many/all scenes\n\nThe authors presuppose many ambitious properties of LLMs/VLMs e.g \" general reasoning capabilites\" (L88). these are still open areas of research and evaluation - LLM reasoning can fail on examples that are trivial for humans. \n\nQuality Justification (technically sound?): lacks valid evaluation results, lacks sufficient qualitative results, lacks any discussion of cost. \n\nClarity Justification (clearly written?) 3. fairly clear, especially once one reads the prompts in the supplement. short excerpts of these could be moved to the main paper. \n\nSignificance Justification (impactful for community? likely to build on ideas?): Generating useful simulation environments for robotics is potentially very significant. But it is impossible to predict adoption given unknown cost.\n\nOriginality (new insights?):  CoT or DPO are now standard practice in using LLMs, I do not find the nature of their use in this paper insightful. The novelty of the work is a set of structured prompts for layout specification, eliciting standard LLM chain-of-thoughts, and/or rewards. LLMs with structured outputs, that are prompted to perform CoT, and/or rewards from another LLM prompt, are prevalent.\n\nQUESTIONS:\nHow were the prompts of Figure 3 selected? Are the results shown random samples (e.g. with a pre-determined seed = 0 for each method, best-of-1)? Are the Fig3 results representative in quality of what I might see if I ran your method for seeds 0..5 on identical prompts as prior work? (e.g. the starwars office prompt of Holodeck Fig 1)\n\nHow can we be assured of the usefulness of larger samples of scenes for downstream use (by humans, in simulators or other graphics) ? Please comment on the validity of quantitative results (see above). I estimate it will difficult to address without a tightly controlled user study (randomized, >7 participants, well crafted survey) which is beyond the scope of a rebuttal.\n\nWhat is the mean/min/max CPU runtime, API credit usage, and/or GPU of the method under various settings? Is this new cost-performance tradeoff appealing for downstream users?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}