Title: SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
Venue: NeurIPS 2025
Accepted: yes
Keywords: 3D Scene Synthesis

Reflective Agents
Abstract: Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that \model not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a language-model-based agentic framework to generate both common and open-vocabulary room types. It dynamically selects tools and iteratively refines the scene based on feedback from language models, and it achieves better results than baselines on both human evaluation and language-model evaluation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The framework unifies a lot of existing scene synthesis methods and improves on them.\n- The paper is easy to follow with a lot of figures to help understand.\n\nWeaknesses\n- My overall impression is that the pipeline can be done through simple optimization without language models. From figure 5, it is hard to see the “iterative” part (maybe except for the move of the table). It seems the pipeline is just about coarse to fine object placement, similar to Infinigen.\n- In figure 4, because these scenes are all different, a naive approach could just be to let the model sample a lot of different scenes and (automatically) cherry-pick one. Because the other approaches do not do cherry-picking, it is easy to outperform them. So it would be best to have more restricted comparisons.\n- The evaluation metric is mostly language-model-based. So it is likely “the blind leading the blind.” Though there are human evaluations, they are limited by the number of participants.\n- In table 1, the criteria might be too strict, for example, Infinigen is classified as not “Real”.\n- The controllability of the method is limited, at least in the experiment. The prompt is only the room type.\n\nQUESTIONS:\n- Can the author provide real “iterative” examples where the model rebuilds part of the scene to improve each of the metrics: visual realism (Real.), functionality (Func.), layout correctness (Lay.), and scene completeness? Otherwise, it is hard to assess the significance of the approach.\n- Can the author provide experiments on using the proposed framework to refine scenes generated by baseline methods, so that the scenes under comparison look mostly similar but improved?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents SceneWeaver, an LLM-based system which integrates many asset libraries and generation tools to perform open-vocabulary language-conditioned generation of indoor 3D environments. SceneWeaver uses multimodal language models to iteratively suggest improvements (\"reflection\") and choose tools to use to add assets, add physical constraints, or correct errors detected by the LLM.\n\nSTRENGTHS AND WEAKNESSES:\nQualitative results:\n- strength: the qualitative results provided are compelling, especially offices and gyms which are challenging (due to objects not attached to walls, e.g. grid of desks or treadmills) and are not supported in prior tools e.g. ProcThor, Infinigen \n- weakness: lacking a larger controlled qualitative sample. ATISS and Infinigen both provide 30+ random qualitative samples in their supplement. The paper would be stronger if it included such a sample, ideally seeds=0,1,2 for each language prompt (so we could have seen the variance due to random seed) and with outputs for other methods for the same prompt or scene category. \n\nTool use:\n- strength: the work integrates an impressive variety of object and scene synthesis tools, which I believe takes nontrivial engineering effort\n- weakness: it is unclear in the main paper which tools use what underlying woreks. paragraph L199 should be significantly expanded in the main paper to make the technical implementation of these tools clear. \n\t- The work uses assets from Objaverse, 3D-Future, Infinigen \"depending on the tool\" (L204) but doesnt state how these choices are made (even in A.3). L75 states tools are used \"based on their respective strengths\" - how does the system know or encode these strengths?\n\t- It is unclear how object resizing works - are the assets scaled up/down (unrealistically?) or does it use infinigen parameters? \n\t- A.3. specifies the arrangement constraint system, but it extremely closely resembles Infinigen's solve_state.json. \n\t- The paper main paper could specify this all carefully (ideally a table with columns \"tool name\", \"role\", \"our contribution\", \"use of existing works\") and expand paragraph L199. \n\nMinor issue: Table 2/3's `#OBJ, #OB and #CN` metrics are gameable\n- These metrics are easily saturatable (many works score 0.0 error), and in theory can be satisfied by a weak baseline (e.g. a shape-packing algorithm that places objects without intersection but with no regard for realism or even gravity). \n- `#OB` and `#CN` are necessary but not sufficient for use in robotics sims. e.g. objects could be floating or topple over in the sim, but wouldnt violate these metrics. Some objects in the demo video would topple in this way.\n- A better metric would be \"what is the sum of total pose changes when we initialize a simulation and let objects settle?\". This would subsume `#OB` and `#CN` (since intersections cause large movements), but would also measure other bad phenomena (object topples over, object drops to the floor). https://arxiv.org/abs/2405.20510v1 and similar use metrics along these lines I believe. \n\nTable 2/3's use of GPT-4 as a judge of \"Realism\", \"Functionality\" and \"Layout\" is invalid for SceneWeaver, (even despite that \"LLMs as judge\" is now a common paradigm for other work). \n-  The SceneWeaver reflection system uses GPT-4 (the exact judge model, mentioned on L178) to actively iterate against the exact \"real/func/layout\" validation scores (confirmed in Appendix Table A2) . If the method accesses the validation scores during generation, then in my view they are invalid as validation scores.  This is similar to reporting the train loss of a deep neural network, or training on a validation set. If the comparison setup of Tab2/3 says this is fair game, ATISS should be allowed to generate 1000 samples and return whichever GPT-4 says is best, before then being judged by GPT-4.\n- A premise of the work is that LLMs need significant scaffolding (e.g. SceneWeaver) in order to correct deficiencies in spatial reasoning. However, the work then uses GPT-4 with minimal scaffolding as quantitative evaluation. \n- A more minor issue: Using natural language prompts as the definition of \"functionality\" is imprecise. Functionality could have a precise geometric definition, e.g. what affordances are preserved (% of sofas/seating that can be accessed from the front, or % of percent of ovens/cabinets that have space to open without obstruction). These are directly important for downstream tasks. I recognize this is a minor and very stringent criticism however, as most prior works do not evaluate this.\n\nThe authors do provide a human user study of 5 participants, which partially addresses my concerns in Tab2/3. In my view, this needs to serve as the main evaluation of the system. Therefore, it would benefit from more participants and a stronger study design to reduce variance. Specifically the user study would ideally:\n- (1) clearly define what evaluators should look for in terms of \"real\", \"functional\" and \"layout\" (currently there are no instructions to participants what these mean - see appendix A7)\n- (2) provide an evaluation of layout which specifically says to ignore asset / rendering quality (similar to Infinigen-Indoors) \n- (3) show multiple outputs for each generation system, to evaluate which _distribution_ of results is better (evaluating individual samples will be high variance) \nOverall, the human user study shows much more minor gains over Holodeck than the GPT-4-as-judge study, which lends credence to the idea that LLMs-as-judges may not be valid for this paper.\n\nThe work does provides insufficient description and argument on the cost of the system. AppendixC-L76 states it is \"minutes to hours\" but this could mean 24 hours. The work uses GPT-4, so it also has a precise dollar-cost of APIs used per scene. CPU cost, GPU cost, and API cost should each be reported as a mean, min and max, or even reported for individual qualitative results, or means reported in Tab2/3 for each method if available. Along with this, I would expect an argument that the cost is worthwhile for the improved results (e.g. \"+X minutes is acceptable for embodied sims because it can be amortized over Y minutes of sim usage\"). If the cost is very very high, it would be rigorous to also evaluate against baselines on an equal-cost setting (fewer reflection steps for SceneWeaver, or maybe increase DiffuScene denoising steps / run Infinigen with very large annealing steps). \n\nIn summary, I believe the work does produce impressive scene arrangements, but would appreciate controlled and more numerous qualitative results. My major concerns are that (1) Table2/3 metrics should not be presented as primary evidence of the work's usefulness and (2) the work does not report detailed cost, does not weigh cost in any comparisons, and does not argue for the acceptability of its cost.\n\n\"Quality\" Justification: technical soundness - LLM-as-judge is not technically sound, requires additional evaluation (qualitative and quantitative). some claims in Tab1 are potentially excessive (\"large-scale\", \"real\"(ism?), \"accurate\" - all are subjective and may apply to baselines)\n\n\"Clarity\" Justification: good except for description of how underlying works are used. \n\n\"Significance\": potentially very useful as a robotics / sim tool. usefulness depends on cost. \n\n\"Originality\" Justification: LLM \"reflection\" or tool use are not original, and i don't believe their application to scene arrangement is highly original, unless I am misunderstanding details of the \"tool cards\".\n\nQUESTIONS:\nAre the existing qualitative results random and non-selected? What is the variance in visual quality if I evaluated random seeds 0...5 with the exact same bedroom prompt?\n\nWhat is the technical contribution of the tool implementations and standardized tool cards? How do the tool cards expand on a normal LLM-tool-use setup e.g. describing the tools to the openai api? The presence/non-presence of tools is ablated, but _standardized and extensibility_ is listed as a contribution (L74). \n\nWhat parts of the \"executor\" (L202) are built by the authors vs taken from existing tools?\n\nI would appreciate any discussion or further justification of Tab2/3/6 quantitative results (see above). In my view, the LLM-as-judge results should be removed from the paper.\n\nWhat is the mean/min/max SceneWeaver cost in terms of CPU-hours, GPU-hours and API credits? What are the costs for some of the examples scenes in the main paper? What are the realism metrics if SceneWeaver is restricted to the runtime of the baselines? How do we know that the cost-realism tradeoff of SceneWeaver is favourable for downstream tasks (robotics/games) compared to baselines?\n\nI believe the work is promising and would argue for acceptance if the authors address my concerns on evaluation and cost.\n\nLIMITATIONS:\nRuntime is listed as a limitation (which is good to report) but requires further details discussion (see above).\n\nNo \"potential negative impacts\" are mentioned but I do not see this as significant.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a system, Sceneweaver, for generating simulable single room scenes from open language prompts with \"reason -> act -> reflect\" strategy. The system primarily leverages a central planning MLLM with mixtures of LLM,MLLM,VLM and some model-based algorithms as \"tools\" to initialize and iteratively refine a scene. These tools scaffold the iterative process and provide clear direction to the central planner about what to refine and how to do so. The tools are broken into \"initializers\", \"microscene implementers\", and \"detail refiners\" with a few of each type implemented, though more could be added. The authors validate their results against SotA systems with quantitative analysis and small scale user study. They also demonstrate a potential robotics simulation application by loading the generated scenes into a physics simulator for teleoperated manipulation.\n\nSTRENGTHS AND WEAKNESSES:\nI like the approach. Iterative language model planning with tool use is a clear strategy and the authors executed on this approach to improve on SotA metrics. I do think some more details could be included and more limitations could be addressed, particularly on the diversity of results and anti-physical content. Also, if robotics is a primary target application it would be good to try the system in task contexts to validate that it generalizes beyond simple open prompts. A user study was nice to see and results were clear, but more than 5 participants would be nice. Overall I'm recommending a weak accept but would consider stronger support with some more details added.\n\nPros:\n- open-language input\n- focus on physically interactive scenes targets robotics simulation well\n- closed loop iterative refinement  vs. monolithic\n- pre-defined \"tools\" inject the hand-designed or model-based organized elements which MLLMs can leverage more openly\n    - the \"secret sauce\" is the combination and implementation of these tools\n- human study to validate the results\n- Captures hierarchical support relationships between objects\n- extensible with more tools\n- supports multiple input modalities (2D guided vs. data-driven)\n- as models improve, so should results\n- because tools have language feedback, it can be understandable compared to quantitative optimization algorithms (e.g. based on statistical priors)\n- object relationships are considered explicitly (e.g. chairs facing the desk)\n\nCons:\n- single room scale\n- lots of prompt engineering\n- unrealistic scaling/placement for functional items (e.g. the gym layout with vertically stacked treadmills and squashed elliptical bikes.\n    - At the end of the day, function dictates parameter limits. You can't scale an elliptical machine and a chair must be large enough to fit a person. These hallucinations limit the effectiveness of the final scenes.\n- Large generation times (up to hours) limits utility in practice\n- minor (out of scope): no articulations or considerations - e.g. drawers and drawers need to open\n- 5 users study with 20 scenes each seems a bit lightweight\n- No task-based specifications (yet?) - interesting to see that robotics simulation was the target here, but task doesn't manifest in examples. I suppose a prompt could include the task like \"generate a kitchen scene with dishes next to the sink for dish washing\", but it would be interesting to see that demonstrated if so.\n\nQUESTIONS:\n- We know there isn't contact between objects from the metrics, but what about excess space? If simulation is run on the scenes, how much shifting occurs in the states? Are object placements stable?\n- How do we iterate on a system like this? Since the system relies so heavily on the mixture of tools and their implementations, what metrics would we use to evaluate its improvement when we add or remove individual tools?\n  - Can we evaluate the impact/effectiveness of a single tool in isolation?\n- \"the secret sauce\" - looks like we're missing details about the implementation of the tools themselves. Beyond the tool cards and scattered discussion we're lacking concreteness on these details.\n  - e.g. how exactly is the \n- We see \"functional diversity\" as a focus for this project, but we don't see any analysis of diversity specifically mentioned. Is this feasible to add for a small prompt domain? How much visual and content diversity can one achieve with a fairly precise prompt?\n\nLIMITATIONS:\nPrimary limitation claimed by the authors is the generation time and complexity.\n\nLikely, other limitations related more to the specific mixture of tool ingredients present in this iteration rather than a fundamental system limitation given that tools can be expanded. However, they could mention: \n- limited complexity of relationships - mouse+keyboard+monitor is a good start, but real robotics tasks will require much more and \"well aligned\" content\n- hallucinations of realism - e.g. \"squashing\" of functional items like exercise equipment or blocking of articulable items like drawers.\n- clearly no structural elements are considered (windows, doors, stairs) and rooms are always rectangular\n- functional access does not seem to be validated: can an agent reach the furniture or objects to use them?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis submission introduces SCENEWEAVER, a reflective agentic framework that aims to unify scene synthesis paradigms through tool-based, iterative refinement. The framework employs an LLM-based planner to iteratively build a 3D scene via a suite of extensible scene generation tools. It performs self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. SCENEWEAVER brings the reason-act-reflect paradigm to 3D scene synthesis and the authors claim that it is the first agentic framework for this purpose. They perform experiments on open-vocabulary scene synthesis to demonstrate its effectiveness compared to existing methods with respect to visual realism, physical plausibility, and instruction following. They also provide ablation studies to provide mechanistic insight into the proposed framework.\n\nSTRENGTHS AND WEAKNESSES:\nThe strength of this submission lies in the integration of the synthesis tools (both generation-focused and evaluation-focused). Because these tools are seamlessly integrated in the framework, the ReAct-style reasoning and planning pipeline for 3D scenes becomes tractable. And because LLMs are now driving the synthesis, open-vocabulary becomes a strength of the generation process. While self-reflection (or at least consistency checks) in 3D is not wholly novel, connecting these established paradigms in a meaningful way through tool use is compelling. That being said, without the ability to review the code, the extensibility of the framework remains unclear. \nThe two weaknesses of this submission both lie in the evaluation. First, leaning on VLM-as-a-judge for visual and semantic evaluation seems problematic as the same model is being used as both agent and judge (this is setting aside that VLMs are known to struggle with spatial reasoning). While it has become common practice, I would like to see stronger citations than 2 preprints as a basis for adopting it.   Second, it is unclear to me that the chosen metrics for physical evaluation are desirable (specifically the number of objects in a scene). Furthermore, claiming best performance when there is a tie (as seems to be the case often with out-of-boundary objects (#OB), and collided object pairs (#CN)) seems incorrect.\n\nQUESTIONS:\nAs alluded to in the weaknesses section, I believe the manuscript would improve with some attention to the evaluation. Is there a justification for why a higher average number of objects in the scene (#Obj) is better? Could #Obj be used to weight #OB/#CN (as it stands to reason there are more opportunities for errors given a higher object count)?\nI like the inclusion of the human study, especially the pair-wise comparison. It would be powerful to add some statistical weight to this aspect of the evaluation by increasing the sample size and demonstrating additional statistics like inter-annotator agreement.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-03"}