Title: OptiScene: LLM-driven Indoor Scene Layout Generation via Scaled Human-aligned Data Synthesis and Multi-Stage Preference Optimization
Venue: NeurIPS 2025
Accepted: yes
Keywords: Indoor scene layout generation

LLM-based

DPO

3D

Synthetic Data
Abstract: Automatic indoor layout generation has attracted increasing attention due to its potential in interior design, virtual environment construction, and embodied AI. Existing methods fall into two categories: prompt-driven approaches that leverage proprietary LLM services (e.g., GPT APIs), and learning-based methods trained on layout data upon diffusion-based models. Prompt-driven methods often suffer from spatial inconsistency and high computational costs, while learning-based methods are typically constrained by coarse relational graphs and limited datasets, restricting their generalization to diverse room categories. In this paper, we revisit LLM-based indoor layout generation and present 3D-SynthPlace, a large-scale dataset that combines synthetic layouts generated via a `GPT synthesize, Human inspect' pipeline, upgraded from the 3D-Front dataset. 3D-SynthPlace contains nearly 17,000 scenes, covering four common room types—bedroom, living room, kitchen, and bathroom—enriched with diverse objects and high-level spatial annotations. We further introduce OptiScene, a strong open-source LLM optimized for indoor layout generation, fine-tuned based on our 3D-SynthPlace dataset through our two-stage training. For the warum-up stage I, we adopt supervised fine-tuning (SFT), which is taught to first generate high-level spatial descriptions then conditionally predict concrete object placements. For the reinforcing stage II, to better align the generated layouts with human design preferences, we apply multi-turn direct preference optimization (DPO), which significantly improving layout quality and generation success rates. Extensive experiments demonstrate that OptiScene outperforms traditional prompt-driven and learning-based baselines. Moreover, OptiScene shows promising potential in interactive tasks such as scene editing and robot navigation, highlighting its applicability beyond static layout generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OptiScene, an indoor scene layout generation method based on the LLM. The authors start by constructing a large-scale indoor scene layout dataset, which contains about 17,000 scenes and covers four room types. Based on the dataset, they develop a two-stage training strategy, including supervised fine-tuning and multi-turn direct preference optimization, to produce structured layout representations from instructions. The results demonstrate the superiority of OptiScene over the prompt-driven and learning-based baselines. They also extend OptiScene to handle other relevant tasks, such as scene editing and robot navigation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The curated large-scale dataset, 3D-SynthPlace, would be valuable for indoor layout generation.\n- Two stage training of SFT along with DPO achieves layout generation effectively. Experimental results show that OptiScene is the SOTA method in this task.\n- The introduction of the two-stage DPO framework, including human consensus alignment and geometric plausibility enhancement, makes the generated layouts closer to human preference and more geometrically valid.\n- Paper writing is clear and easy to follow.\n\nWeaknesses:\n- Some notations and equations are confusing. For example, in Eq. 2, the LLM outputs $X$, which represents the input instruction. Besides, Does the so-called \"bounding box\" refer to the component size? It seems very easy to confuse with the concept of \"layout\" (i.e., $c$).\n- The paper lacks some implementation details. For example, it is unclear how the object retrieval and layout editing (data construction, data format) are achieved.\n- Introducing high-level semantic reasoning into the model output seems not effective.\n- The ablation studies seem not thorough. Only two quantitative results are reported in this part.\n- OptiScene generates layouts based on object types, but in real scenarios, users may also have requirements for the relative relationships between objects.\n\nQUESTIONS:\n1. Could the authors elaborate on the implementation details of the object retrieval module? Error accumulation maybe a key issue in current framework. That is to say, if the retrieval accuracy is not high, the final output will not conform to the user instruction.\n2. Following up on the previous question, current evaluation metrics lack the assessment of the consistency with user instructions. How well can the output layout meet the user needs?\n3. It Tab. 6, the introduction of semantic reasoning brings little performance improvement. Do the results mean that the semantic reasoning in SFT is not that necessary? Could the authors explain the results? It would be great if the authors could share the quantitative results of \"SFT+DPO1+DPO2\" without \"Reasoning\".\n4. How is the layout editing task achieved using the layout generation model? Either the dataset construction and format is not clear.\n5. The FID and GPT-4o Ratings are missing in the ablation studies.\n6. In DPO-stage II, the authors add noise to GT layouts to produce negative samples. What is the noise level used? Do the authors study the effect of different noise levels? Are the negative samples produced by large noise better than those produced by small noise? It would be great if the author could have a discussion about this.\n7. Does the current model support layout generation based on object relationships (e.g., the bed is to the east of the desk)? Users sometimes also specify the relative positions of the objects rather than just the types of the objects.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces OptiScene, an LLM-based framework for preference-aware indoor layout generation. Unlike prior prompt-driven approaches that struggle with controllability and physical understanding, and learning-based approaches that suffer from scarce data sources leading to a lack of expressiveness, OptiScene addresses these concerns by post-tuning an LLM for the spatial design task using the proposed large-scale 3D-SynthPlace dataset.\n\nThis is achieved in two steps: The SFT stage introduces a high-level semantic reasoning step before layout generation, while the two-stage DPO pipeline helps align layout generation with human preferences and physical constraints.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1)\tThe proposed pipeline attempts to combine the strengths of prompt-driven and learning-based methods, and further extends them by aligning the generation process with human preferences.\n2)\tThe paper introduces a synthetic indoor scene dataset called 3D-SynthPlace, which covers a wide variety of layouts along with semantic summaries and scene descriptions, contributing to the field.\n3)\tThe proposed approach demonstrates potential for different applications such as scene editing and object-centric navigation.\n\nWeaknesses:\n\n1)\tMissing details for SFT: Although Section 3.3. discusses the SFT process, some crucial details are missing from the text or require further clarification:  a) L195-196: How is the supervision provided to minimize the discrepancy between predictions and ground-truth layouts, including both the semantic summary and the structured output? b) How is the user instruction U converted into the JSON specification X? c) Generally, object retrieval from the dataset is performed using the Euclidean distance between the predicted bounding boxes and the bounding boxes of the dataset assets, as well as category information. However, Equation 1 appears to use object descriptions for retrieval. How this is achieved is not clear.\n2)\tValid overlapping bounding boxes: DPO Stage 2 and the way prompts are constructed (L196–197) are designed to avoid object collisions. However, real-world scenarios often include cases where valid overlapping bounding boxes naturally exist (e.g., a chair underneath a desk). How does the method handle these cases? Is it possible to include them as challenging positive examples?\n3)\tPossible bias in the training data: a) Despite the advanced filtering, the training dataset relies on Holodeck’s predictions and may inherit its biases. b) During DPO training, negative samples are generated by the SFT model itself, which could lead to similar error patterns and limited diversity.\n4)\tLack of ablations on user instruction complexity: No evaluation is performed showing how the free-form instructions which may exhibit high complexity and ambiguity affect the generations.\n\nQUESTIONS:\nAll of my questions are listed in the weaknesses section, and I may adjust the rating if they are well addressed. Specifically:\n\n1) Some parts of the method, particularly the SFT process, require further clarification.\n2) Additional experiments could be conducted to evaluate the method’s behavior in challenging cases (e.g., valid overlapping bounding boxes) and with complex, ambiguous free-form instructions reflecting real-world scenarios.\n3) Potential bias in the training data should be discussed, and generalization experiments should be conducted.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to the problem of scene layout generation from a natural-language description of a set of indoor objects. It introduces **OptiScene**, an LLM‑based framework fine-tuned through SFT and RL (DPO [22]). Additionally, the manuscript also introduces **3D-SynthPlace** a novel dataset constructed from Holodeck[35], significantly extending 3D-Front[9] dataset.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n* The paper is well-written and explain clearly the motivation behind the proposed solution. It offers a clear view of the current modules and contributions, and effectively evaluates the proposed method/solution in a suitable benchmark with valid baselines for scene layout generation.\n\n# Weaknesses\n* The current manuscript fails at explaining and discuss the limitation of the current solution. For instance, more complex scenes (specifically more objects) and un-seen items (objects/items no presented during training).\n\nQUESTIONS:\n* Q1: in L302, Object-Centric navigation experiments are presented. The experiment description defines a command \"find a <object> in the room\" to execute a robot exploration. Results are presented in Tab 5. Can the authors elaborate on how the robot/agent explores the scene to find the object? If the robot does not use the proposed **OptiScene** why this experiment is evidence that the proposed solution allows the downstream task of navigation?\n\n* Q2: The ablation study in Tab. 6 indicates that prompting with a high-level description (reasoning) significantly boosts performance (as pointed out in L314). I’m curious whether other formats of high-level input, such as a JSON file with some centroids or even visual inputs (image in BEV), would have a similar effect or not. How might the results change with these different formats?\n\n* Q3: How the proposed solution performance with more complex scene layouts? For instance 5, 10, 20, 30 objects. Is there any limitation in this matter.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes to fix the gap between prompt driven LLM approaches which are costly and learning based methods which are constrained. First it proposes a new dataset-- 3D-SynthPlace to expand the 3D-Front dataset with synthetic samples generated with Holodeck and then filtered with a human in the loop system. The dataset not only doubles the size of the dataset but also increases the diversity of the scenes to include kitchens and bathrooms. The authors train an LLM on this dataset to output a structured JSON which captures the scene with scale and position of the objects. Rather than solely training the LLM using SFT, the authors choose to use RL based DPO fine-tuning for greater alignment with human preference by comparing against perturbed versions (collision) of the scene. The method is compared against 5 baselines and an ablation of the SFT vs RL is provided.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n* The paper is well-written and all necessary details are provided. \n* The results are quite strong, both qualitatively and quantitaively. The evaluation setup apart from LLM as a judge is robust and the ablation shows good gains in performance when tuning with RL. The additional user study paints a more complete picture.\n* The method does not need to perform constrained optimization solve like FlairGPT or Holodeck which makes it computationally more feasible. \n* Lower token cost compared to prompt driven methods & multi-agent frameworks\n* The RL based fine-tuning with petrurbed scenes is ineteresting and lowers errors and object collisions (object overlap rate)\n\n## Weaknesses\n* Related works such as Holodeck and FlairGPT also take into account the door & window positions in the room and walls + objects on walls but this method misses out on this vital detail. Small objects make the scene much richer and provides a more complete layout of the room. Given that the authors use Holodeck for creating the synthetic dataset, I find the omission of small objects a limitation. The authors do not discuss this omission from Holodeck. \n* It's well known in the community that SFT memorizes and RL generalizes [SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training, Chu et al], therefore the contribution is not particularly unique but it might be the first scene layout method to apply this (is this the case?). Nevertheless, I think the authors should mention this in their work since it's not a novel observation. \n* The LLM as a judge (GPT-4o Ratings) for layout evaluation is not robust. It's well known that LLMs fail to provide reliable ratings in a 1-10 scale, compared to their text summaries which are considerably better.  \n\n\n## Neutral\n* The dataset could have been a nice contribution given the human-in-the-loop system for filtering poor layouts. However, since the method disposes small objects, the dataset contribution becomes weaker.\n\nQUESTIONS:\n* Does the structured JSON proposed here bring any advantage over other very similar representations proposed in related works?\n* I'm very much on the borderline, please address weaknesses\n\nLIMITATIONS:\nDoor and window positions are missing in the generated layouts. It should be discussed in limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}