SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
Authors: Yandan Yang
Baoxiong Jia
Shujie Zhang
Siyuan Huang
Venue: NeurIPS 2025 poster ()
Keywords: 3D Scene Synthesis
Reflective Agents
Primary area: applications

Abstract: Indoor scene synthesis has become increasingly important with the rise of Embodied AI, which requires 3D environments that are not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained to fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, we present SceneWeaver, a reflective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SceneWeaver employs a language model-based planner to select from a suite of extensible scene generation tools, ranging from data-driven generative models to visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop reason-act-reflect design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and open-vocabulary room types demonstrate that \model not only outperforms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking a step toward general-purpose 3D environment generation.

SCENE WEAVER: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent
Yandan Yangl
Baoxiong Jia 18 Shujie Zhang 1.2
Siyuan Huang 4.
State Key Laboratory General Artificial Intelligence, BIGAI
Tsinghua University
https://scene- weaaeeeereewweerr...mm.. github..hh......... io/
Refiner: Rotala the tabics to avaid crowding
Initializer:
Implementar:
objects
SceneWeaver
Implementer:
Executor:
Physical Optimization
Make sure collaiion
Refiner: Remove imgraper bethroom sink.
Office
Restaurant
Gym
Kitchen
Figure 1: Overview of SCENEWEAVER, WEAVER. reflective agentic framework built standardized and extensible tool interfaces that unifies the strengths of existing scene synthesis methods produce visually realistic, physically plausible, instruction-aligned 3D scenes.
Abstract
Indoor scene synthesis has become increasingly important with the rise of Em- bodied AI, which requires 3D environments that not only visually realistic but also physically plausible and functionally diverse. While recent approaches have advanced visual fidelity, they often remain constrained fixed scene categories, lack sufficient object-level detail and physical consistency, and struggle to align with complex user instructions. In this work, present SCENE WEAVER, a re- flective agentic framework that unifies diverse scene synthesis paradigms through tool-based iterative refinement. At its core, SCENEWEAVE employs language model-based planner to select from suite of extensible scene generation tools, ranging from data-driven generative models visual- and LLM-based methods, guided by self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. This closed-loop ceason-act-refles design enables the agent to identify semantic inconsistencies, invoke targeted tools, and update the environment over successive iterations. Extensive experiments on both common and vven-vocabulary room types demonstrate that SCENEWEAVE not only outper- forms prior methods on physical, visual, and semantic metrics, but also generalizes effectively to complex scenes with diverse instructions, marking step toward seneral-purposs 3D environment generation.
Equal contribution. PProeect lead. Corresponding authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
3D scene synthesis [1, 2, 3, 5, 6, 7, 8, 9, 10, 11] has been a standing research topic in computer vision and graphics, primarily focused on generating visually realistic 3D environments for applications such as interior design, virtual content creation, and gaming asset creation. With the recent rise of embodied artificial intelligence (EAI), the scope of scene synthesis has naturally expanded to accommodate new functional demands [12, 13, 3]. Beyond achieving visual realism, scenes are now expected to be physically interactable within simulators and precisely controllable in response to task-specific user instructions, particularly in constructing tailored environments for training and evaluating embodied agents. These extended requirements pose significant new challenges for 3D scene synthesis.
Despite rapid progress, existing methods fall short of holistically addressing the requirements for realistic, controllable, and physically plausible scene synthesis, as summarized in Tab. 1. Rule-based systems [12, 14] ensure physical validity through hand-crafted constraints, but lack extensibility across diverse scene types and offer limited controllability due their rigid, manually defined logic. Data-driven generative learning methods [15, 31. while more flexible, are constrained by the scarcity of nigh-quality, scene-level 3D datasets (e.g., 3D-Front [16]). As result, they typically produce visually realistic scenes within pre-defined categories but generalize poorly to novel scene types or layout instructions. Methods based on Large Language Models (LLMs) approaches [7, 8, 9, 10, 11] offer stronger open-vocabulary understanding and semantic flexibility, yet often struggle with spatial reasoning and 3D awareness, resulting in physically implausible rearrangements. Collectively, these limitations highlight that no single approach is sufficient to meet the combined demands of realism, physical plausibility, and controllability. This motivates the need for comprehensive and adaptable scene synthesis framework capable of synthesizing high-quality 3D scenes.
Inspired by recent advances in L-Le- agents, which demonstrate strong reasoning and planning capabilities in complex tasks, recent works in 3D scene synthesis have begun to move beyond monolithic approaches by decomposing the generation process into sequential compositions of modular synthesis components, forming multi-step pipelines coordinated by LLMs. A common strategy starts with generating coarse, scene-level layouts through interaction with LLMs [10, 8, 9, 7], followed by progressive refinement using pre-trained 2D generative models or Multi (MLLMs) for asset generation [17, 18], object placement [19, 20, 21 and texture inpainting [11, 22]. While these pipelines leverage both the specialization of individual models and the semantic flexibility of MLLMs, they remain largely "static", i.e., their planning and execution are governed by fixed prompts and hard-coded module invocation logic over limited set of synthesis tools. This design overlooks the potential to couple reasoning with adaptive decision-making based on generation feedback, and the ability to seamlessly integrate diverse synthesis tools through unified interface. As result, these systems fall short of enabling self-refining and extensible agents, leaving the full potential of multi-modal foundation models underutilized.
LLMs
To address the aforementioned challenges, we propose SCENE SCENEWEAVER, WEAVER, a reflective agentic frame- work that enables MLLMs to synthesize 3D scenes in a self-refining manner through set of casily extensible tool interfaces. Specifically, SCENEWEAVER consists of two core components: 1) standardized and extensible tool interface that abstracts diverse scene synthesis methods into mod- ular tools operating at different levels of generation granularity; 2) a self-reflective planner that dynamically selects tools and iteratively refines the scene by reasoning over feedback from previous generations, while applying the planned modifications and enforcing physical plausibility with a physics-aware executor. This framework enables dlosed-loop. feedback-driven scene evolution, where the agent identifies areas for improvement, invokes appropriate tools, and updates the scene under physical constraints. Extensive experiments show that SCENE WEAVER achieves new state-of-the-arr across a broad range of scene types and open-vocabulary instructions, demonstrating strong visual realism, physical plausibility, and precision in instruction following. We also provide ablation studies showing that the self-refining design is critical to achieving high-quality scene synthesis and that integrating diverse tools leads to significant performance improvement compared to monolithic approaches. In summary, our contributions are as follows:
We propose SCENEWWAAVRR the first reflective agentic framework for 3D scene synthesis, enabling MILMs to iteratively refine scenes through feedback-driven planning with modular tools.

---

Table 1: Comparison of different scene synthesis methods. A single approach is not sufficient meet the combined demands of realism, physical plausibility, and controllability, which motivates the need for comprehensive and adaptable scene synthesis framework.
Previous Work
Physical
Plaus.
Object
Small
Open
#Room Type
Real
Accumate
Large Scale
CAD Source
Developing Platform
Method
KTISS [15]
DiffuScene [2]
3
3D FUTURE
Model-based
Scenee
Infinigen [14]
Prochho| (12)
/
5-
x
Generated RoboTHOR
Blender A12 12-THOR
Rule-based
MetaScenc [19]
30+
/
Mixed Behavior
ACDC [20]
Unlimited Unlimited
V
OmniGibson
Vision based
Archileat
Mixed
LayomGPPT [9]
FUTURE Mixed
Holodack [10]
A12-THOR
AnyHome
Unlimited
Generated
LLM-based
I.Design [7]
Objaverse
Layou(VLM [8]
Objaverse
SCENEWEAVER
Unlimited
Mixed
IsaaeSim
Unified
SCENEWEAVE introduces a comprehensive reason-act-reflect paradigm that formalizes the plan- ner's decision making, reflection, and action protocols, along with a standardized and extensible tool interface for synergizing diverse scene synthesis methods based on their respective strengths. Extensive experiments open-vocabulary scene synthesis demonstrate that SCENEWEAVER outperforms existing methods both visual realism, physical plausibility, and instruction following. We also provide meticulously designed ablation studies to highlight the effectiveness of the proposed reflective agentic framework.
2 Related work
3D Indoor Scene Synthesis 3D indoor scene synthesis is typically formulated as a layout prediction
task, where objects are represented by 3D bounding boxes and semantic labels [16, 15, 8]. Data-driven generative models| [15, 2, 3]. trained on datasets like 3D-FRONT [16]. learns realistic but coarse scene layouts, constrained by the limited variety and level of detail of scenes in the dataset. To address this limitation, recent work leverages language and 2D foundation models to provide missing priors on scene types and fine-grained details. LLM- methods 8, 10, 11] combine textual prompts with rule-based systems [12, 14] to generate diverse scenes, but often suffer from hallucinations and the poor spatial reasoning capability of LLMs. Meanwhile, methods based on 2D foundation models improve scene detail and spatial coherence through conde--onditioned generation [17, 18] or real-to-sim conversions [20, 19]. However, they remain limited by the capability image generation models and challenges in 2D-to-3D lifting, exhibiting semantic or physical inconsistencies under complex scene generation instructions. Overall, no existing paradigm sufficiently balances realism, physical plausibility, and controllability. To this end, propose SCENEWEAVER a unified and extensible self-reflective agentic framework that integrates the complementary strengths of existing approaches for high-quality 3D scene synthesis.
Spatial Reasoning of MLLMs Recent works have explored using the reasoning and generative abilities of MLLMs for 3D scene synthesis. To address their limitations in spatial reasoning, these methods often incorporate structured constraints and external logic to enhance physical plausibility. Some approaches apply rule-based constraints as post-processing to correct implausible object placements [10], while others adopt multi-agent or role-based decomposition to reduce hallucinations and improve coherence [7). Additionally, efforts have been made to guide generation through scene- aware tools, such as programmatic layout representations [8] or geometric reasoning modules [23]. Although these systems MLLMs with reasoning chains and post-optimization mechanisms, they typically rely on fixed toolsets and predefined constraints, limiting their ficxibility and extensibility, In contrast, SCENE SCENEWEAVER WEAVER isdesigned to support diverse and extensible set of tools through standardized interface, enabling dynamic tool selection and composition via a reflective planning mechanism for reasoning-drit driven 3D scene synthesis.
LLM--base Agentic Framework A growing body of work leverages LLMs as autonomous agents for complex tasks across domains such as scientific discovery [24], clinical decision- [25], and visual reasoning [26]. As L.MSs reasoning capabilities gradually advance, the focus has shifted from narrow sask-specific agents to general-purpose agentic frameworks [27, 28, 29, 30] that
3

---

Tool Cards
Final Scene
Query
STOP
Iniliialtze
Planner
Sones
PhyScone
Think
Reflection
Aedt Uleid ruprore
Implementer
Action
Feedback
Assets
LLN
Executor
+P hysical optimize
3D FUTURE Notscole
Refiner
Objavarse Liningenn
Tool Rerrove Object
Updata Rotatin
Add Rolation
Collision
Out bound
Relaton
Figure 2: The SCENE WEAVER pipeline. Following reason aet--aatt--eee reflect paradigm, SCENEWEAVER iteratively
refines scenes by integrating the strengths of diverse scene synthesis tools.
coordinate multiple specialized tools solve complex problems collaboratively. Recent work [31] has demonstrated that the extensibility and planning capabilities of LLMs, i.e., the ability to flexibly integrate diverse tools and coordinate them effectively, are crucial for solving complex reasoning tasks and lead to significant performance gains. However, these insights on agent development remain largely underexplored the context of 3D tasks. Motivated by its relevance to 3D scene synthesis, SCENEWEAVE draws on advances in LLM-based agentic frameworks and adopts the OpenManus [30] platform implement an agentic framework, with a particular emphasis on extensibility of tools and the reason-act-react paradigm for 3D scene synthesis.
3 The SCENE WEAVER Framework
agentic framework that enables LLMs to perform feedback feeddack-guided, self-reflective 3D scene synthesis using a diverse set of scene synthesis tools. The SCENEWFANT FEAVR framework comprises two key components: 1) standardized tool interface that organizes the majority of existing scene synthesis methods into modular tools categorized by their synthesis granularity (Sec. 3.1): 2) self-reflective planner that dynamically selects tools, iteratively refines the scene based on feedback, and performs physics-based optimization to enhance physical plausibility. An overview of SCENEWEAVER is provided Fig. 2.
In this section, present the design of SCENEWEAVE
Before describing each component, we formalize the overall problem setup. Given a user query q E Q and a tool set D [d.) SCENEWEAV aims synthesize 3D scene sy through T iterative refinement steps. Each scene state &, is represented by both 3D layout information and also 2D renderings from selected camera views (as illustrated in Sec. A.1). At each step t [1..., T). the self-reflective planner receives a reflection ft-1 including quantitative scores and explanatory justifications assessing the quality and instruction alignment of the previous scene 8t-1. Based this feedback, the planner selects a tool d, E to refine, and the physics-aware executor applies the refinement and performs physical optimization produce the updated scene s,. A new reflection v, is then computed for st, and the process repeats.
3.1 Standardized Scene Synthesis Tool Interface
Tool Catalog As summarized in Tab. 1, existing 3D scene synthesis methods vary widely in their design and focus. To leverage their complementary strengths within unified framework, we introduce a standardized tool interface that abstracts each method as a modular synthesis tool. These tools categorized according their synthesis granularity:
Scene Initializer: This class of tools generates full-scene layouts and serves as the starting point for synthesis. We categorize initializers into three types: 1) data-driven generative models [15, 2, 3]. which offer scalable generation learned from human-designed indoor scene datasets but are limited pre-defined scene types; 2) real-to-sim methods [20, 19] that create digital twins cousins

---

mibstzer Reatzaim MeasBrone
Indt wlzer Model PlySoene
Task desoription: You designer,
3D bendiwly design telrw
spotial aptiization. Your mission
scue moaa any mon cyn
User demant: Design me hundry room.
Shaps
Asallabl Tools,, Inttalzoos, implomenters, Refiners.
cpniia snkiraliny
Mamery
Imparmenter 20 gaited Archiler
the rolation
machine.
Inplamenter OPT
Refiner: VLM S.M-Uppie Updato Rotation Success.
Tool Selection Exsecution Reeult
infemmmerr 2D guided MCDC
Scane Represeristton
Rendered image and leyoul
into
Refiection score & suggestion
Bas Cas da srocp suil
Object number &, Colletion Out room:
Bireigte preegtte metl Accente ln resetie bostlen bor sicac
Real Fandiional Layout Completinn 5.
dooor daily supplios makes the room unfinisted.
aijene
step
Ralner LLM- Rumeve Object
reflection of
check previcus problems have solved.
scheed The rocation probiem
Refier RuMe Add Relation
2. Find the most
problem to salv this stap. The complen pleston score low Add bunnrry objects the shat
Refinor: VLM Updale Rotation
Instsipste:: toe run moee
Lar aoprepette tools with 6-1 confidence score, and choose
Hos cow dtace
octect
sec's
one tool for this stop. Use implementer: LLM: GPT to add
oojerss
Bea Lase de iepeeie sprticl orpanccation silgpiig alpers
Cleary y scolein the expectation and suggeet nedt staps auauoossss weallas ls adjustments. innot, try another strllegy
Tirm Iegrnio ruieet ald Ean
Objects ahould be widedd
Isyour and proceed
vobile
Figure 3: A visualization of standardized tool interfaces and the reflective planning process. The self- reflective planner leverages diverse tools to first correct the misoriented laundry machine and then enhance scene details by adding small objects to the shelf (right).
of realistic scenes, providing detailed high-quality scenes but with limited diversity and scale; 3) LLLM-based [9. 10, 8, 7] that enable open-vocabulary and flexible generation from natural language, but often exhibit semantic or physical inconsistencies due limited spatial reasoning.
Microscene Implementer: This class of tools adds micro scene details (e.g.. small objects placed desks or shelves) that are often missing from whole-scene synthesis methods. We consider two types of implementers: 1) LLM--based tools that generate microscene layouts conditioned on local context (e.g.. placing a keyboard and monitor on a desk), offering semantic diversity but prone to spatial placement errors (e.g., misaligned or backward-facing objects); and 2) 2D-guided tools [17, 20), which synthesize reference images of microscene regions using pre-trained 2D generators, then mapping corresponding 3D assets to the predicted layout. While still constrained by the spatial reasoning capability of 2D models, the 2D-guided tools enhances visual realism and relative spatial coherence between objects.
Detail Refiner: While previous tools synthesize scenes at various granularities, they often introduce errors such as object misplacement or implausible configurations. Refiner tools address these issues by enforcing constraints and refining object poses. First, we extend on rule-based scene synthesis methods [12, 14] and use LLMs to convert user queries into relational constraints that guide object placement. Second, to compensate for layout generators that neglect object orientation and scale, we incorporate dedicated tools to refine objects' full 6D pose (location, rotation, scale). Finally, an LLM--based remover identifies and eliminates semantically incorrect or severely misplaced objects.
Standardized Tool Cards To ensure flexible integration of new tools into SCENEWEAVER we define standardized tool cards that guide the planner in deciding when and how to invoke each synthesis method based on their specialized strengths. Examples are shown in Fig. 3. Each tool card contains mandatory fields, including tool description, applicable scenarios, usage constraints, and required input parameters. We also inc lude example usage and tool-specific strengths to help the planner select the most appropriate tool based on user queries or iterative feedback. For initializer tools. supported room types are listed to reflect model-specific limitations and enable the agent to infer room types from queries when evaluating tool applicability. This modular design ensures seamless integration, extension, and replacement of scene synthesis methods without modifying the overall agentic framework.
3.2 Feedback-driven Self-reftective Planning
we first define the process for generating self-evaluated feedback over synthesized scenes. Specifically, given a generated scene 8t, we invoke an MILM (e.g., GPT-4) to produce reflection i, comprising two components: 1) physical metries, including collision scores, room boundary violations, and object count and diversity; and 2) perceptual metrics, including visual realism, functionality, layout
Reflection Generation To support self-reflective planning in
5

---

coherence, alignment with the user query, and scene completeness. In addition to scalar scores, the MLLM generates natural language justifications and improvement suggestions as input to the planner. This feedback forms a core reasoning signal for the planner in the subsequent step, enabling to assess tool effectiveness and adapt its strategy accordingly. If the feedback indicates abnormal degradation (e.g., sharp drops in quality or constraint violations), the planner can roll back and replan the current step.
Self-reflective Planning Given the user query a tool set D, and memory of previously selected tools, generated scenes, and reflection feedback m4 -t- 0e-(:-1), where / de- termines the length of memory, the planner in SCENE WEAVER determines the most appropriate refinement action. Leveraging context-award function-calling capabilities in LLMs, the planner first summarizes the current context (i.e., memory) and identifies the most critical problem to address at step 1. It then ranks candidate tools by suitability and confidence, selects the most promising tool d, E DD and generates tool-specific instructions (e.g., "populate empty tables with contextually relevant objects") Tool confidence scores are dynamically adjusted based on past performance, i.e., failures reduces confidence, and repeated failures trigger replanning by reprioritizing refinement targets and selecting alternative tools. Our self-reflective planner is built on the OpenManus plat- form [30], following the ReAct-style [32] reasoning and planning pipeline. We provide illustrative examples in Fig. and more detailed prompts in Sec. A.2.
Since most tools described in Sec. 3.1 operate on 3D bounding box layouts. a physics- aware executor is required to replace these drafts with concrete 3D assets and enable both physical post-optimization and accurate evaluation of physical metrics. To this end, we build our executor on top of Infinigen [14] and Blender. At each iteration f, the executor loads the previous scene and the layout modifications proposed by tool d,, then retrieves and replaces object instances with 3D meshes from a collected asset pool combining Objaverse [33], 3D-Future [34], Inifinigen [14], etc., depending on the tool. To ensure spatial consistency with relationships or constraints generated by detail refiner tools, the executor also adjusts object placements to satisfy relational constraints (e.g., aligning chairs to face desks) produced by the detail refiner. It then performs a fixed number of thysics-basee optimization steps to resolve collisions and boundary violations. We provide additional implementation details in Sec. A.5.
Physics-aware Execution of Plans
4 Experiment
In our experiments, we aim to answer the following questions: Q1: How does SCENEWEAVER perform compared to existing data-driven and open-vocabulary scene synthesis methods? Q2: How does the reflective agentic framework behave during the iterative scene refinement? Q3: How effective each module in SCENE and how critical are they to overall performance?
Settings We quantitatively evaluate WEAVER against existing methods under two primary settings: 1) common room types, where large-scale human-designed datasets support direct data- driven learning, and 2) open-vocabulary scene generation, following [8], which evaluates generation across diverse room type descriptions. In the common setting, models are evaluated based on the average score over 10 scenes each for the living room and bedroom categories. In the open- vocabuacyy setting, evaluation is based on the average score over scenes for each of room types, using the prompt "Design me troom_types queries to assess SCENE CCEENEWEAVER 's fine-grained controllability over scene generation, with details provided in Sec. 4.3. For all settings, we set the maximum number of iterations in SCeneWeaVER to 10. The memory length is set to to avoid hallucination. We provide additional experimental details in Sec. B.
as the user query. We also include a setting with complex
Baselines For the common settings, we compare with data-driven scene synthesis models including ATISS [15], DiffuScene [2], and PhyScene [3]. For these three methods, we train the model over the 3D-Front [16] dataset following the conventional learning evaluation schemes. We also compare with state-of-the-ane art -pen-vocabulary 3D scene synthesis methods including LayoutGPT [9], Holodeck [10], and I-Design [7] on both the common and the open-vocabulary setting. As LayouGGP was originally limited to bedrooms and living rooms, we adapt it to open-vocabulary room types by modifying its prompts and constraints. To evaluate the final scene quality, we retrieve assets from Objaverse [33] using OpenShape [35] text embeddings following [7].

---

Table 2:
Quantitative comparison common room types between SCENEWEAVER and existing scene synthesis methods. For L..M-hased methods, use "Design <room terom__ype>"""" as the user query.
Bedroom
Living Room
Method
Physcis
Visual & Semantics
Physcis
Visual & Semantics
#
#08
PO 1
Red.
Fusc. 1
Legt
Cong.1
405 1
POW
Red. 1
Funt.
Lest
Comp.
ATISS [15]
3.9
0.5
0.6
7.4
7.1
6.6
4.2
7.8
0.1
0.7
5.8
5.3
6.4
3.7
DiffuSeene [2]
3,5
0.1
1.1
6,5
7.0
6.7
3,6
6.9
0.5
1.2
5.5
4,9
5.2
3.5
PhySoene [3]
3.3
0.3
5.7
5.7
4.0
8.0
0.7
5.2
5,3
5.1
3.3
LayourGPT
5.4
1.0
1.3
7.5
8.1
6.7
4.2
8.4
2.8
6.4
5.8
5.2
3.6
Holodeck [10]
32.2
0.0
38.5
8.6
9.1
7.8
6.2
23.0
0.0
5,3
8.9
9.3
7.6
8.1
1-Design [7]
9.6
7.6
9.7
8.4
8.9
7.7
5.9
Ours
14.0
0.0
0.0
9.2
9.8
8.4
9.4
17.3
0.0
0.0
9.1
9.5
8.0
8.7
Table 3: Quantitative comparison on open-vocabulary generation between SCENE WEAVER and existing methods, We report the average score across 8 scene types to evaluate overall model performance.
Bathroom
Children Room
Gym
Method
m8g
PGI
Rai
Bec
Lay.
Deep
K2k3
92a
Red.
Fax.
1ay.
Corp
nCB
ICE
Red.
Tax.
Lay.
Goup.
LiyouGPT
7.7
1.3
1.0
8,3
9.3
7.7
6.0
7,3
1.0
0.7
6.3
8.0
6.0
4.0
6.7
0.7
0.0
6.7
6.7
5.7
3.7
Holodock
12.0
0.0
1.7
7.7
6.7
7.0
5.3
13.7
0.0
2,0
7.5
7.5
6.5
5.5
20.3
0.0
5.3
9.7
9.3
6.7
6.0
I-Design
0.0
0.0
7.4
7.2
11.3
0.0
0.0
8.3
5.5
12.0
8.4
5.2
Ours
19.7
0.0
0.0
9.0
10.0
23,0
9.0
10.0
29.7
10,0
Meeting Room
Office
Restaurant
Method
may
Rail
Tay
BCS
Red
Fax
lay
Coep
ITR
Rel
1ay
Eway
LayoutiPr
7.3
1.0
0.7
4.0
3.0
5.3
2.0
7.3
0.3
0.0
6.7
7.7
6.3
4.0
7.01
0.3
1.7
3.3
2.3
4.7
2.0
Holodeck
27.0
0.0
0.3
9.0
10.0
8.0
7.0
27.0
0.0
4.7
7.0
6.3
4.3
4.0
35.0
0.0
12.3
5.1
1.3
4.3
3.7
I-Design
18.7
5,3
0.0
6.0
4.5
588
4.3
11.7
0.0
0.0
8.0
9.0
6.8
5.4
27.7
0.0
0.0
6.2
5.2
5.2
4.0
0.0
0.0
9.0
10.0
N.7
N8.0
7.0
7.3
Waiting Room
Kitchen
Average
Method
Ital
hec
Lay:
Usep
8733
RG1
Red.
Tax.
Lay.
Uiep
Rea.
Fix..
Lay.
Urmp
LigonGPP
6.3
0.0
0.3
6.7
5.7
6.0
4.0
1.7
1.3
1.3
5.7
6.3
4.7
3.7
7.3
0.7
0.7
6.0
6.1
3.7
Holodeck
24.0
0.0
3.7
8.3
9.3
6.7
5.7
20.0
0.0
1.3
7.3
6.3
6.3
4.3
22.3
0.0
3.9
7.7
7.5
6.2
5.2
L-Design
10.7
0.0
0.0
6.6
6.4
5.8
42
11.7
0.0
0.0
6.5
6.8
5.5
3.5
14.5
0.7
0.1
7.1
7.0
6.2
4.7
Ours
25.7
0.0
0.0
9.0
10.0
8.0
7.7
34.7
0.0
0.0
9.0
9.3
7.3
7.7
36.5
0.0
0.0
88
9,4
7.7
8.0
Metries For all quantitative evaluations, we evaluate models using physical, visual, and semantic metrics following [3, 7J. For physical evaluation, we report the average number of objects in the scene (#Obj), out-of- cut-oo-boundary objects (#DB), and collided object pairs (#CN) as the main metrics assess physical plausibility and realism of the scene. For visual and semantic evaluation, we report scores for visual realism (Real.), functionality (Func.), layout correctness (Lay.), and scene completeness (Comp.) as indicators of visual quality and semantic coherence with the user query. Following [7, 8), we use GPT-4 to assess these metrics, providing with top-down renderings of the generated scenes and the user query as input. To further assess objects' stability in simulation, we evaluate the shift. distance in supplementary.
4.1 Scene Generation for Common Room Types
We provide quantitative evaluation results for the living room and bedroom in Tab. 2. Results show that SCENEWEAVER achieves state-of-the-art results across most metrics, outperforming both data-driven generative models and open- vrcebuocbbulrry models. Notably, Holodeck slightly surpasses SCENE in the number of objects (#0bj=32.2). However, we argue that this is primarily due to the inclusion of randomly placed objects, often lacking rationality in object placement. Data- driven methods tend to generate scenes with fewer objects, as their training datasets are largely composed of large furniture items. Consequently, their visual and semantic scores are also lower due to the limited quality and diversity of the training dataset. Interestingly, we observe that data-driven methods outperform LayouGGPT on physical metrics, suggesting that relying solely on LLM-based generation is insufficient for ensuring physical plausibility. In contrast, our LLM-based agentic framework, empowered by reflection and physics-based optimization, achieves zero physical errors. which is comparable to pipelines that enforce hard constraints during optimization (e.g., Holodeck). qualitative comparison of generated scenes provided in Fig. 4.
4.2 O Open-vocabulary Scene Generation
We present quantitative evaluation results in Tab. 3. The results show that SCENEWEAVER sig- nificantly outperforms existing open- vocabulary scene generation methods across all eight tested room types. It achieves an average object count of 36.5, notably higher than other approaches, and
7

---

ATISS
Diffuseene
Physcene
SeeneWeaver (Ours)
Redreos
Lining Room
LPI
Helodeck
I-Design
SeeneWeaver (Ours)
Remoaront
Meeting
Kinchoe
Figure 4: Qualitative comparison between SCENEWEAVER and existing methods on both synthesizing
common room types and open-vocabulary room types, SCENEWEAVER produces scenes with improved visual realism and finer-grained detail compared to prior methods.
also achieves significantly better visual and semantic scores. More importantly, WEAVER accomplishes these improvements while strictly satisfying physical constraints (i.e., achieving zero collisions and out-of-boundary violations). This highlights the effectiveness of the reflective planner in both improving semantic coherence with the user query, scene diversity, and in fixing physical implausibilities in the iterative refinement process. We provide qualitative comparisons against other methods in Fig. 4 to further demonstrate the superior visual realism and semantic coherence of scenes generated by SCeneWEAVER Overall, both quantitative and qualitative metrics confirm that SceneWeave consistently outperforms existing methods on open-vocabulayy scene generation, highlighting the effectiveness of our reflective agentic framework.
4.3 Additional Analyses
Ablation on Agent Design We conduct an ablation study on agent design by evaluating variants of SceNeWEAvER on the average of three kitchen scenes following the open- -pca-vocybulary scene generation setting. Specifically, we consider the following variants: 1) removing the reflection module (wo Reflection), 2) removing the physical optimization module (w/o Phys. Optim.), and 3) replacing iterative reflection with single-shot multi-step planning (Multi-step Plan). As shown in Tab, 4, removing the reflection module results in notable drop in semantic quality, while omitting physical optimization significantly harms physical plausibility. Additionally, compared to the multi- step planning variant, SCENE WEAVER achieves superior visual and semantic performance. This highlights the importance of iterative reflection, as single-pass planning often generates globally inconsistent or locally infeasible layouts by failing to account for context-dependent refinements.
Effectiveness of Tool Cards To evaluate the impact of different tool types, we ablate the use of specific subsets from our tool set during scene generation. As shown in Tab. 5, adding or removing particular tool types significantly affects performance across all metrics, demonstrating the importance of tool diversity and validating the design of our standardized Specifically, we observe that modifier tools help align scenes with functional requirements and improve layout coherence, but may reduce object count (16.3 vs. 23.0) and completeness (5.0 vs. 5.7) by removing redundant items. In contrast, implementer tools excel at enriching scenes with appropriate details, enhancing realism, functionality, and completeness. The full combination of initializer, implementer, and modifier tools yields the highest performance, highlighting the complementary strengths of diverse tools in 3D scene synthesis.
achieving
8

---

Tablc 4: Ablation Agent Design.
Table 5: Ablation Effectiveness of Tools.
Method
40bj
40n
ecr
Resl
Finc.
Lay
Comp
Tools
a0ej
Func.
Comp.
w/o Reflection
25.0
0.0
0.0
8.0
8.3
6.3
6.3
Init.
23.0
7.7
7.0
5.7
w/o Phys. Optim.
27.3
0.7
2.0
8.3
9.3
6.7
7.7
Init-Modifier
16.3
7.7
8.3
6.3
5.0
Multi-step Plan
29.3
0.0
0.0
8.3
7.7
7.0
7.3
Imit+mmplem.
34.3
8.0
8.3
6.3
7.3
Ours
34.7
0.0
0.0
7.3
7.7
Full
34.7
9.0
9.3
7.3
7.7
lmmenter the shet
Retteen
Refher
implementer: washing maciins
Firshh and tecsum
on the shatl
in the stail
Query: A laundromaa with machines, Add washing supplics machine. Add other related objects, baskets, washthub the
1ST banio soene
Fxyv eci
Add pepecis
Adki doporatirs
Frodio room and fastorn.
andid crowded.
supperer.
well
Query: A bedroom rich of fumiture,, decoration on the wall, and small objects.
Figure 5: Iterative refinement SCENEWEAVER given complex user queries. SCENE WEAVER progressively incorporates detailed elements specified in the user instruction, demonstrating its ability to iteratively refine and generate high-quality, instruction-aligned 3D scenes (best viewed with zoom-in).
When presented with complex user instructions, SCENEWER W WEVER leverages iterative refinement better follow detailed require- ments, particularly in object count, small object placement, and overall scene layout. We provide two qualitative examples of the iterative refinement procedure Fig. to illustrate this capability.
Iterative Refinement in SCENEWEATE
with Complex Queries
Human Study To further assess the quality of scenes generated by SCENEWEAVER, duct a human study with twenty participants.
Table Human Evaluation Results.
con-
Method
aübj
Real.
Func.
Lay.
Comp.
Each participant evaluates five scenes randomly from the vocabulayy scene generation set- ting using the metrics following Sec. 4.2. As shown in Tab. 6, SCENE EAVER consistently outperforms baseline models across all dimen- sions. And the Human LLM alignment is shown in supplementary. Additionally, we conduct pairwise comparison study, where participants indicate their preference and check diversity be-
LayoutGPT
5.94
5.83
6.21
5.26
5.99
I-Design
7.20
6.65
6.57
5.73
6.79
Holodeck
7.86
6.70
7.35
6.67
7.45
Ours
9.30
8.80
8.85
8.55
8.98
Table 7: Preference and diversity over other models.
Method
w/ I-Design
w/ Holodeck
w/ LayoutGPT
Preference
94.30%
91.40%
87.40%
Diversity
95.60%
98.90%
90.00%
tween scenes (3 for each room type) generated by SCENEWEAVER and baseline methods. Results in Tab. 7 show that BEAVE is preferred in nearly 85% of cases. These findings underscore the strength of SCENEWEAVER in producing visually and semantically coherent indoor scenes.
5 Conclusion
In this work, we present SCENEWEAVER, a reflective and extensible agentic framework for 3D scene synthesis that integrates diverse scene synthesis paradigms through standardized tool interfaces and iterative feedback- driven refinement. By adopting a act-reectt- paradigm, SCENE WEAVER enables LLM-based planner to dynamically select and invoke appropriate tools, guided by multi- modal self-evaluation of physical plausibility, visual realism, and instruction alignment. This closed- loop design allows achieving superior performance across both common and open-vocabulary scene settings. Extensive experiments and human evaluations validate the advantages of our approach in producing high-quality, functionally coherent, and semantically faithful 3D scenes. We believe SCENEWEAVER represents a step toward general-purpose, controllable 3D environment generation, with broad implications for Embodied AI, simulation, and interactive agents.
FEVER to effectively decompose and correct complex generation tasks,

---

References
[1] Siyuan Qi, Yixin Zhu, Siyuan Huang, Chenfanfu Jiang, and Song-Chin Zhu. Human-centric indoor scene synthesis using stochastic grammar. In CVPR, 2018. 2
[2] Jiapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Diffuscene: Denoising diffusion models for gerative indoor scene synthesis. In CVPR, 2024.
3, 4, 6,
[3] Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. In CVPR, 2024. 2, 3, 6,
[4] Qiuhong Anna Wei, Sijie Ding, Jeong Joon Park, Rahul Sajnani, Adrien Poulenard, Srinath Sridhar, and Leonidas Guibas. Lego-net: Learning regular rearrangements of objects in rooms. arXiv preprint Stti:::300.096299 2023. 2
[5] Guangyao Zhai, Evin Pinar Örek,, Dave Zhenyu Chen, Ruotong Liao, Yan Di, Nassir Navab, Federico Tombari, and Benjamin Busam. Echoscene: Indoor scene generation via information echo over scene graph diffusion. arXiv preprint arXiv:2405.00915. 2024. 2
[6] Zhifei Yang, Keyang Lu, Chao Zhang, Jiaxing Qi, Hanqi Jiang. Ruifei Ma, Shenglin Yin, Yifan Xu, Mingzhe Xing, Zhen Xiao, et al. Mmgdreamer: Mixed-modality graph for geometry- controllable 3d indoor scene generation. arXiv preprint arXiv:2502.05874 2025. 2
[7] Ata Çelen, Guo Han, Konrad Schindler, Luc Van Gool, Iro Armeni, Anton Obukhov, and Xi Wang. I-design: Personalized II interior designer. arXiv preprint DxXi::2444002838, 2024.
2, 2, 5, 7
[8] Fan- Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutvlm: Differentiable optimization of 3d layout via vision-language models. CVPR, 2025. 2, 6,
[9] Weixi Feng. Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuchai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and generation with large language models. NeurIPS, 2024. 2, 3, 6,
[10] Yue Yang. Fan-Yur Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjie Liu, Chris Callison-Burch Mark Yatskar, Aniruddha Kembhavi, and Christopher Clark. Holodeck: Language guided generation of 3d embodied ai environments. In CVPR, 2024. 2, 3,
[11] Rao Fu, Zehao Wen, Zichen Liu, and Srinath Sridhar. Anyhome: Open-vocabulary generation of structured and textured 3d homes. In ECCV, 2024. 2.
[12] Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Jordi Salvador, Kiana Ehsani, Winson Han, Eric Kolve, Ali Farhadi, Aniruddha Kembhavi, and Roozbeh Mottaghi. ProcTHOR: Large- Scale Embodied AI Using Procedural Generation. In NeurIPS, 2022. Outstanding Paper Award. 3.
[13] Mukul Khanna*, Yongsen Mao*, Hanxiao Jiang, Sanjay Haresh, Brennan Shacklett, Dhruv Batra, Alexander Clegg, Eric Undersander, Angel X. Chang, and Manolis Savva. Habitat Synthetic Scenes Dataset (HSSD-200), An Analysis of 3D Scene Scale and Realism Tradeoffs for ObjectGoal Navigation. arXiv preprint. 2023. 2
[14] Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, Zeyu Ma. and Jia Deng. Infinigen indoors: Photorealistic indoor scenes using procedural generation. In CVPR, 2024. 3,
[15] Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fidler. Atiss: Autoregressive transformers for indoor scene synthesis. NewrIPS, 2021. 2,
6,
10

---

[16] Huan Fu, Bowen Cai, Lin Gao, Ling- Xiao Zhang, Jiaming Wang, Cao Li, Qixun Zeng, Chengyue Sun, Rongfei Jia, Binqiang Zhao, et 3d-front: 3d furnished rooms with layouts and semantics. In ICCV, 2021. 2, 3,
Ling-Xiao
[17] Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang, Tsun-Hsuan Wang, Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting. In NeurIPS, 2024. 3.
[18] Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deging Sun, Ming-Hsuan
and Ming- Hsuan Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.07207 2024. 2.
[19] Huangyue Yu, Baoxiong Jia, Yixin Chen, Yandan Yang. Puhao Li, Rongpeng Su, Jiaxin Li, Qing Li, Wei Liang, Zhu Song- Lang-Chun, Tengyu Liu, and Siyuan Huang. Metascenes: Towards automated replica creation for real-world 3d scans. In CVPR, 2025. 2, 3,
[20] Tianyuan Dai, Josiah Wong, Yunfan Jiang, Chen Wang, Cem Gokmen, Ruohan Zhang, Jiajun Wu, and Li Fei-Fei. Automated creation of digital cousins for robust policy learning. In CoRL, 2024. 2, 3, 5,5
[21] Lu Ling, Chen-Hsuan Lin, Tsung-Yi Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming-Yu Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXiv:2505.02836, 2025. 2
[22] Dave Zhenyu Chen, Haoxuan Li, Hsin- Ying Lee, Sergey Tulyakov, and Matthias Nießner. Scenetex: High-quality texture synthesis for indoor scenes via diffusion priors. In CVPR, 2024.
2
[23] Ian Huang, Yanan Bao, Karen Truong, Howard Zhou, Cordelia Schmid, Leonidas Guibas, and Alireza Fathi. Fireplace: Geometric refinements Ilm common sense reasoning for 3d object placement. CVPR, 2025.
[24] Andres M Bran, Sam Cox, Oliver Schilter, Carlo Baldassari, Andrew D White, and Philippe Schwaller. Chemcrow: Augmenting large-language models with chemistry tools. arXiv preprint arXiv:2304.05376, 2023.
[25] Samuel Schmidgall, Rojin Ziaei, Carl Harris, Eduardo Reis, Jeffrey Jopling, and Michael Moor. Agentelinic: a multimodal agent benchmark evaluate simulated clinical environments,
2024. 3
[26] Yushi Hu, Weijia Shi, Xingyu Fu, Dan Roth, Mari Ostendorf, Luke Zettlemoyer, Noah A Smith, and Ranjay Krishna. Visual sketchpad: Sketching as a visual chain of thought for multimodal language models, 2024. 3
[27] Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen Ilm applications via multi-agent conversation. arXiv preprint arXiv:2308.08113 2023.
[28] LangChain. I. langchain.
ai ai/anngchannn,n,n, 2024. 3
[29] OpenAl. Function calling openai. https::/plattorm www.oppeennnneeeeeeoo guid//////
function-
function-calling, calling, 2023a. 3
[30] Xinbin Liang, Jinyu Xiang, Zhaoyang Yu, Jiayi Zhang, Sirui Hong, Sheng Fan, and Xiao Tang. Openmanus: An open-source framework for building general ai agents, 2025. 3.
[31] Pan Lu, Bowen Chen, Sheng Liu, Rahul Thapa, Joseph Boen, and James Zou. Octo- tools: An agentic framework with extensible tools for complex reasoning. arXiv preprint arXiv:2502. /271, 2025. 4
[32] Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting language models. arXiv preprint arXiv:22 10..0669, 2022. 6
11

---

[33] Matt Deitke. Dustin Schwenk, Jordi Salvador, Luca Weihs, Oscar Michel, Eli VanderBilt, Ludwig Schmidt, Kiana Ehsani, Aniruddha Kembhavi, and Ali Farhadi. Objaverse: A universe of annotated 3d objects. In CVPR, 2023. 6
[34] Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, and Dacheng Tao. 3d-future: furniture shape with texture. IJV,, 129:3313-332 2021.
[35] Minghua Liu, Ruoxi Shi, Kaiming Kuang, Yinhao Zhu, Xuanlin Li, Shizhong Han, Hong Cai, Fatih Porikli, and Hao Su. Openshape: Scaling up 3d shape representation towards open-wooee apen-world understanding. NeurIPS, 2023,
[36] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. In CVPR, 2024. A1l
[37] Ziniu Hu, Ahmet Iscen, Aashi Jain, Thomas Kipf, Yisong Yue, David A. Ross, Cordelia Schmid, and Alireza Fathi. Scenecraft: An Ilm agent for synthesizing 3d scene as blender code, In ICML., 2024. All
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, the abstract and introduction include the main claims.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of the work the appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
13

---

Justification:
This paper makes no theoretical claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details ensure reproducibility.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer: [Yes]
Justification:
We will open-source the code and dataset upon acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /Codeeuomiisseeeeeeee
/nips. wwtpccccpppccceecccccccccccccclt / ssissssoiccyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide details on experimental settings in the experiment section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We randomly sampled 10 scenes for common room types and scenes for open-vocabulary scene generation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide details on resources used in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: : http///neuissspeeeeeee cc htse//ublic/Eteieee
httppp/lic/hhccccuuilleeleleeeee ctw.iiiscccieeeececccccc ines?
Answer: [Yes]
Justification:
We follow the NeurlPS code of ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact of our work in the appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The method proposed poses no obvious risks at hand.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We appropriately credited the authors when using existing datasets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: This paper leverage existing 3D assets to generate new 3D scenes, however, these new scenes are mainly treated as experimental results.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [Yes]
Justification:
We provide details on human studies in the appendix.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [Yes]
Justification:
We follow the IRB procedures for human studies.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
18

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [Yes]
Justification: We use LLMs as one of our major model basis for orchestrating different tools for scene generation.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: : //tpr//nss.////// theurips.cc/Coffeeee eee.ceeeeeeeee2//////// 20LM/LLM) for what should or should be described.
19

---

A Details of the SCENEWEATS Framework
A.1 Scene Representation
As mentioned in method, the 3D scene at each step is represented by a combination of 3D layout data and 2D rendering. The details are illustrated in Fig. Al. On the left, we show top-down rendering of the scene Blender, which helps align the visual representation with the coordinate- based layout shown on the right. To enrich the spatial understanding, we mark the image with X, Y, and coordinate axes at the coordinate origin and 2D projection coordinates on (x.y) plane to emphasize the spatial position. Each object is further labeled with its 3D bounding box and semantic category to assist the agent in object recognition. We also mark each object with a 3D bounding box and its semantic label to help agent recognize each object. Since visual language models (VLMs) may struggle with spatial reasoning caaaoninn-particlarrtt particularly object orientation we additionally annotate each bounding box with directional arrow indicating the object's front. On the right side, the layout encodes each object's semantic category as the key, with its location, rotation, and size as values. We also record relational information for each object, including its parent object and the type of relationship.
(0,4
"objects": { 51123552 bed":
bed
"Location": [2.73, 3.43, 0.14],
"rotation": 9.8, 3.14),
(0.3
"size": [2.34, 1.18, 8.82],
"parent": [["reor", ""frroo""""nflee""" "onfloor"), ["roon", "against wott"])>.
dresser
Camp
cand
81122267 desk"
"Location": [2.0, 1.45, 0.14],
00
"rotation": [0.0, -0.0, 1.57], "size": [0.59,
0.78), "parent': [["reon", "onfloor"]]], 33741500 laptog": {
desk
"location": [1.62, 1.53, 0.92).
"rotation": [8.0, 0.8, -1.52). [0.22,0.32,0..4
radiutor
"parent": [['8162267 desk", "ontop"]l),
Figure A1: Example Scene Representation. To convey both visual and logical information of the current scene, we express the scene data in two representations: 1) top-down rendered image I, (left) with coordinate points, axies-arrow and objects' 3D bounding boxes with labels and direction arrow and 2) the objects' layout La (right) including open- open-vocabulary category name, location, rotation, size and relation between objects.
A.2 Self-reflective Planner
provide full prompt to the self-reflective planner and feedback mechanism in Tabs. Al and A2.
A.3
Implementation of the executor
We take Infinigen as the base code of our executor. And we apply multiple modifications on it to fit our agentic framework. Specifically, we update the code to:
update iteratively
interactive blender in realtime with socker
fit each tool rather than generating scene procedurelly
do physical optimization to avoid collision and enhance relation
add 3D marks and 2D top-down rendering after generation
A1

---

Table A1: Prompt for planner.
Prompt for Planner
You are a scene designer, an expert agent in 3D scene generation and spatial optimization. Your mission iteratively design and refine scene maximize its realism, accuracy, and controllability, while respecting spatial logic and scene constraints.
Task description:
prompt. carefully inspect the current configuration and determine the best action to build or enhance the scene structure. You should list all the effective optimization strategy for the next step based solely geometry. layout relationships, and functional arrangement. You must not focus on style, texture, or aesthetic appearance. To achieve the best results, combine multiple methods over several iterations. Start with foundational layout and refine it progressively with finer details.
Note: Given
Available Tools:
(metadata of available tools)
User demand:
(user_demane
Memory of step.-1:
planning ideas)
tool selection & execution results)
[scene representation)
[reflection score suggestion]
Plan for step::
Based on user needs and current status:
Clearly explain the execution results of last step and tool.
According scene information and evaluation result, check previous problems have been solved.
According to evaluation result, which GPT score is the lowest? What physical problem does it have?
Find the most serious problem to solve.
To solve the problem, list all the appropriate tools that can match the requirement for next step with 0-1 confidence score:
You should consider the suggestion from previous conversation to score each tool.
If the same problem has not been solved by last step, you should consider degrade the score of the tool in the last step.
You should carefully check current scene, and you MUST obey the relation of each object. If there is no previous step, init the scene.
For complex tasks, you can break down the problem and use different tools step by step to solve it, but you only choose and execute the suitable tool for this step.
When multiple tools are applicable to solve the user's request, list them with confidence score.
You must choose tool for this step. Clearly explain the expectation and suggest the steps. If there is no big problem to address, or if only slight improvements can be made, or if further changes could worsen the scene, stop making modifications.
A2

---

Table A2: Prompt for Verifier.
Prompt for Verifier
Task
You are given lop-down room render image and the corresponding layout of each object. Your the four criteria listed below. For
task is evaluate how well they align with the user's preferences each criterion, assign score from 10, and provide brief justification for your rating. Scoring must be strict. If any critical issue is found (such as missing key objects, obvious layout errors, or unrealistic elements), the score should be significantly lowered, even if other aspects are fine. Score Guidelines
Score Fully meets exceeds expectations; no major improvements needed.
Score 5: Partially meets expectations; some obvious flaws exist that limit usefulness.
Score Completely fails meet expectations; aspect is absent, wrong, contradicts user needs.
Evaluation Criteria
Realism:
How realistic the room appears. Ignore texture, lighting, and doors.
Good (8- 18): The layout (position, rotation, and size) is believable, and common daily
objects make the room feel lived-in. Rich of daily furniture and objects.
Bad (0-3): Unusual objects strange placements make the unrealistic.
Note: If object types combinations defy real-world logic (e.g., bathtubs in bedrooms),
should be below 5.
Functionality: How well the room supports the intended activities.
Good (8-10): Contains the necessary furniture and setup for the specified function.
Bad (0-3): Missing key objects contains mismatched furniture (e.g., bed in bedroom).
Note: Even missing critical item should lower the score below
Layout: Whether the furniture is arranged logically in good pose and aligns with the user's preferences.
Good (8-10): Each objects in reasonable size, neatly placed, objects the same category are well aglined, relationships are reasonable (c.g., chairs face desks), sufficient space exists for walking, and orientations must be correct.
Bad (0-3): Floating objects, crowded floor, abuormal size, objects with collision, incorrect orientation, large items placed oddly (e.g., sofa not against the wall). Large empty space. Blocker in front of furniture.
Note: If the room has layout issues that affect use, it should not score above 5.
Completion: How complete and finished room feels.
Good (8-10):AII necessary large and small items are present. Has rich details. Each shelf has multiple objects inside. Each supporter (e.g. table, desk, and shelf) has small objects on Empty area
than 50%. The room feels done.
Bad (0-3): Room is sparse or empty, lacks decor key elements.
Note: If more than 50% of the room is blank lack detail, under 5.
User demand |
Rendered Image trendered_image ]
Room layout [layout L.)
Results Return the results following JSON format, comment should short:
"realism"; {
"grade": your grade int,
"comment": "Your comment and suggestion."
"functionality": (...),
"layout": {..., "completion" {...}
A.4 Tool Cards
provide detailed prompts to all tools Tabs. A8-A17. And we list the implementation details of each tool in Tab. A3. During the planning process, the agent is provided with brief description of
A3

---

Figure A2: Robot interacts with the generated SCENEWEAVER in simulation. The first three show the sequences interaction the front view in three different scenes including kitchen, meeting room and restaurant. And the last row shows the side view of the third row. Note the system keeps different materials, such as table in the meeting room has transparent and reflective material.
each tool and uses the function calling choose single tool for each step. Then it will the tool itself and go through the executor to update the scene.
A.5 Physics-aware Executor
Referring to Infinigen, the relation types here include two aspects.
1. Relation between the object and room:
<against_walle wall>: the object's back faces to the wall, and stands very close or exactly on the wall.
Ladde_against_wee
the object's side (left, right, or front) faces to the wall, and stands
very close.
<on_floor>; the object stands the ground.
2. Relation between two objects:
the child object's front faces to the parent object, and stands very close, such as chair and dining table.
Erront_agains
<front_te tt tt__tofffff froftott:: the child object's 's front faces to the parent object's ' front, and stands very close, such chair and desk, coffee table and sofa.
<leftright. ttttt_t____a tt__tt_lleefi too_lefrrgghtt the child object's 's left or right faces to the parent object's 's left or right, and stands very close.
<side_ba balle__________ bi_di::>; the child object's 'sss side (left, right, or front) faces to the parent object's side (left, right, or front), and stands very close.
tock__t____e baack_tobbacke the child object's back faces to the parent object's 's back, and stands very close.
<on_top>: the child object is placed the top of the parent object, such monitor and desk, vase and table.
<inside>: the child object is placed inside the parent object, such book inside shelf.
A.6 How to choose assets dataset
In this project, we choose different asset according to the usage of tool. You can also choose any of them to suit your requirements.
A4

---

Table A3: Tool Overview and Contributions.
Tool Name
Role
Use of Existing Works
Our Contribution
Init MetaScenes
Init scene with
MetaScenes
Choose data & con- vert format
Real2Sim dataset
Init PhyScene
Init scene with pre- trained model
PhyScene DiffuScene ATISS
Choose data & con- vert format
Init GPT
Init scene with GPT
LLM
Prompt engineering Significant changes digital cousin Prompt engineering
Add ACDC
Add tabletop objects
Stable
Diffusion
&
visually
ACDC
Add GPT
Add objects with
VLM
GPT
Add Crowd
Add crowded layout
VLM & Infinigen
Utilize Infinigen rules & design module Prompt engineering
Remove Object
Remove inappropri- Object
VLM
Add Relation
Add relations to ob- jects
VLM & Infinigen
Prompt engineering
& Utilize Infinigen re-
lations
Update Rotation
Fix rotation problems
VLM
Prompt engineering
Update Size
Rescale objects
VLM
Prompt engineering
Update Layout
Update improper lay- outs
VLM
Prompt engineering
MetaScenes: For tool using Dataset such as MetaScenes, we employ its assets directly, since each scene contains several assets with delicated mesh and layout information.
For tool using Model such as PhyScene DiffuScene ATISS, we employ 3D FUTURE, since the model is trained on this dataset.
3D FUTURE:
Infinigen:
For other tools, we use Infinigen's asset generation code to generate standard assets in common categories, such as bed, sofa, and plate. The asset will be generated in delicated rule procedure the scene generation process.
Objaverse: For those categories that are not supported by Infinigen, such as clock, laptop, and washing machine, we employ open- open-vocabulary Objaverse dataset.
B
Experiments
B.1 Additional Experimental Details
The maximum number of steps is set to 10. However, the procedure may terminate earlier if the intermediate results already meet the user's requirements with a high score. The reflection module determines whether continue optimizing or stop.
For asset retrieval, we gather resources from available tools when possible. For instance, when using tools based on data-driven methods, adopt 3D-FUTURE assets. If no assets provided, we first rely on the Infinigen generator to produce standard assets following predefined rules. For more open-vocabulary assets, we refer to OpenShape assets lack a unified initial pose, we calculate their minimum bounding rectangles to identify four side candidates, then prompt GPT to annotate the front- front-facing direction. GPT achieves a high success rate identifying the front side of commonly known objects, though it may fail for more complex ambiguous cases. For the ablation study, we focus the kitchen room type and generate three scenes for the experimental setting.
retrieve objects from Objaverse. In cases where
B.2 Additional Scene Generation Results
We show more visualization results of SCENE WEAVER in Fig. A3. The results of restaurant, garage, and gym confirm that SCENEWEAVE able to arrange multiple objects neatly when the number of
A5

---

Design mesting room.
Design me
Design room.
Design me an office.
Design me gym.
Design me bathroom.
garage with a car in the center. Add work bench and shelf with releted tools.
laundromat with machines. Add washing suppliies on each machine. Add other related objects, such as beskets, and washthub in the room.
bedroom furniture, decoration on the wall, and small objects.
Figure A3: More visualization examples of generated scenes.
the same category is more than three. Cabinet in the bathroom contains objects inside, such roll of paper, since has supporting surface the plane inside. Shelves are equipped with related objects inside (basket in garage and towel in gym). The third row shows some detailed results of complex user queries.
B.3 Simulation in Isaac Sim
We export the generated scenes USD files and load them into Isaac Sim for physical simulation and interactive tasks. Through Apple Vision Pro, we remotely control Unitree G1 humanoid robot to perform object interactions within these virtual environments. As demonstrated in Fig. A2 and our supplementary video, the system supports diverse interaction scenarios across multiple scenes: the first three rows showcase interaction sequences from front-view perspective, while the last row provides a side-view view analysis of the third scene. This pipeline offers three key advantages for embodied AI applications:
figh-fidelity simulation with preserved textures and geometric details.
Robust physical interactions guaranteed by collision-free and ment.
object place-
Task-aligned scene layouts that adapt to diverse EAI requirements through controllable synthesis.
With the combination of these features, we believe SCENEWEAVER enables reliable sim-to-real transfer for robotic manipulation tasks while maintaining visual and functional realism.
B.4
User Study
We invited twenty participants to evaluate the scenes generated by SCENEWEAVER. All participants were volunteers without compensation. We invite them to assess the scenes in two settings.
A6

---

Table A4: Score aligummet. To further confirm the stability of LLM's judgement and human study, we check the alignment between different users as well as LLM
Method
Realism
Functionality 1
Layout t
Completion 1
User-User 0.43
0.42
0.45
0,40
User-LLM
0,46
0,45
0,48
0,55
Fresues this indtor zowne usage or codk from poooy jeccesocc each crow eaitann aqpazze
Compare
Ogroro sandoring qualle)
Pronpe bedgy me dichee.
Kitchen
3
3
12
Dienlly
Rrd
0
0
method
Fundtoall
0
0
0
0
nethoe
Layas
Wh ds ore has paate charaty?
(a) The first setting to score each scene.
(b) The second setting for pairwise comparison.
Figure A4: Example of user study in different settings.
In the first setting, we randomly collected 100 scenes generated by three baseline methods and SCENEWEAVER. Each volunteer randomly assigned 20 scenes along with their corresponding prompts. Participants were asked to rate each scene on a scale from to 10 using the same five metrics described in the experiment. Note that physical metrics such as collision count (#CN) and out-of-boundary objects (#OB) were excluded from this human evaluation, as they are difficult to assess by eye. For each of the five remaining metrics, we provided guiding sentence help participants make consistent and informed judgments. Finally, we aggregated the ratings from all participants and computed the average score for each metric.
In the second setting, we conducted a pairwise comparison study, shown in Fig. A4b. For each baseline method, selected five set of scenes, where each set includes three scenes generated by SceneWeevte and three scenes generated by the baseline method under the same prompt. We increased the generated scene number from to 3 for each method in order to 1) reduce individual variance and 2) check diversity of different generation system. We asked the participants to choose:
Which method do you prefer?
Which method has greater diversity?
We collected votes from all participants and calculated the average preference between SCE- NEWERVER and each baseline method. Results Tab. shows our methods has greater diversity while also stand for higher preference, surpassing other methods large degree.
B.5 Human-LLM Alignment.
Note the evaluation results of human evaluation and LLM score can not be exactly the same, thus slight difference between them is normal phenomenom. To further confirm the stability of these justification methods, we check the alignment between different users and LLM. We convert the users' and LLM's scores different scenes to rankings and calculate Kendall's ' Tau for the
A7

---

Table A5: Shift in simulation. We assess object stability in simulation in Isaac Sim.
Method
>0.Im |
>0.0l
Average Shift
ATISS 35,4%
51.4%
0.356
DiffuScene 26.2%
39.3%
0.190
PhyScene
9.7%
19.6%
0.069
LayoutGPT
39.2%
52.8%
0.477
IDesign 5.0%
11.5%
0.041
Holodeck 17.6%
42.5%
0.113
Ours
1.0%
10.37%
0.011
Table A6: An example of metric improvement during iteration.
Step
Tool
Realism
Functionality t
Layout f
Completion t
Initializer
6
6
2
Implementer
7
6
5
4
3
Refiner
8
7
6
Refiner
8
7
5
Implementer
8
7 8
8
metrics, shown in Tab. A4. The results show strong user-user agreement and user-LLM agreement on different metrics. And we find that user-LLM has higher alignment scores than user-user, which indicates that LLM is more stable than user in evaluation. These alignment results also demonstrate the effectiveness of using LLM for justification in Tab. 3 of the main paper.
B.6 Physical Stability
To assess object stability in simulation, measure:
Shift thresholds:
percentage objects moving >0. Im >0.01m 3s
Average displacement: Mean shift distance (meters)
As shown in Tab, A5, the lowest shift of our methods confirms our great performance the original physical metrics #OB and #CN. Above all, our generated scene remains the most stable in simulation although with the largest object number.
B.7 Metric improvement during iteration.
The overall is self-adaptive rather than coarse-to-fine object placement. The agent in pipeline will find the biggest problem each iteration according to the reflection score and choose related tool to solve the problem. For example, the teaser in Fig. serves obvious example. Here show the metric scores after executing each step in Tab. A6.
The agent chooses the lowest score improve. Specifically, step has the lowest score in completion, which Then in the step 2, the agent chooses the implementer to objects in shelf and improve the completion score from 4 to 6. Now the lowest score is the layout, which is 5, owing to the illogical location of bathroom sink. So in step 3, it uses refiner to remove the bathroom sink, improving the layout score from 5 to 6. So now the lowest score belongs to both layout and completion. In step 4, the agent focuses on the crowding area and decides to use refiner to rearrange the dining tables to ensure adequate space for movement and clear walkways. Then the score of layout becomes 8. The lowest score comes to the completion again. So the agent choose implementer again add objects each table to improve the completion score.
The scores of realism and funetionality will also be improved while we focus on optimizing other metrics. And the agent can fix the problem caused by previous steps, such removing the bathroom sink generated by the previous step. And can utilize the same tool multiple times to improve the quality until it satisfies the demand.
A8

---

Figure A5: Adding room structure including door and windows. show two samples of scene generation with room structures. Each sample includes views from both outside and inside.
Table A7: Improvement on the scenes generated by baseline models.
Method
#Obj f
Realism f
Functionality t Layout f Completion
PhyScene
5
6
PhyScene Ours
11
10
8
LayouGGPT
6
LayoutGPT Ours
20
9
9
8
10
B.8 Impact of a single tool
Tab. 5 shows that the addition of different types of tools (Initializer, Refiner, Implementor) greatly improves the results. To further assess the effect of a single tool, we observe that without the "Update Rotation tool, the model is less sensitive invalid rotations, reducing the "layout" score. Meanwhile, the "Add Crowd" tool lets supporters be densely filled with child objects (e.g.. a shelf packed with books), which improves "realism". If there multiple tools in similar function, adding/removing single tool will not make a big difference. And if the tool has negative effect, the agent with the memory and reflection module will recognize it during iterations and avoid using the tool.
B.9 Room Structure & Single Room Scale
In the main paper, we remove windows and doors to simplify the generation process, though our method can handle such structures and update the scene accordingly. Here show some examples in Fig. A5.
For room scale, we focus single rooms to improve scene quality prompted by the user. The process can repeated to handle multiple rooms. We could also generate multi-room scenes at once with some coordinate transformation.
B.10 Refine scenes generated by baseline methods
Here we refine scenes generated by two baseline methods, PhyScene and LayoutGPT. (IDesign and Holodeck will take longer time in format conversion.) Results show method could greatly improve the baseline method. Actually the baseline method serves as an initializer tool in this setting, and convert each baseline method to tool and merge them into the extensible tool cards.
C Miscellaneous
C.1 Claims in Tab. 1
Here we clarify criteria dicussed in Tab. 1.
Real Defined as realistic layouts considering human daily habits.
Large-scale Methods able to generate infinite scene variations.
A9

---

Accurate Methods with well-defined rules ensuring explicit object relations and strict placement.
For example, Infinigen's realism relies simple rule, texture and rendering, but layout issues remain: (1) monitors may face walls, as placement rules lack direction awareness; (2) random sampling without spatial priors scatter desks. Therefore, we do not consider Infinigen "real".
C.2 Broader Impact
Our work focuses on 3D scene synthesis, aiming to generate physically interactable environments based on complex, user-specific instructions. A key application lies in the development of embodied artificial intelligence, where such synthesized scenes can be used to train agents across diverse tasks. Furthermore, the overall architecture of SCene is grounded in recent L.LM-hased tool-use agent frameworks, positioning it to inspire future agentic systems tailored to specific use cases. This includes the design of task-specialized components such as system prompts and interaction protocols for enhanced application-specta performance. At present, we do not anticipate any immediate negative societal impacts resulting from SCENEWEAVER.
C.3 Resources used
All reported experiments are conducted on a machine equipped with an NVIDIA GeForce RTX 4090 GPU. To generate scene, the time consumption ranges from minutes hours, depending on the iteration number, chosen tools, and crowded status. We use Blender 3.6 to record and render the scene.
C.4 Time Consumption
The time consumption a bit longer due to several reasons. First, different method takes different time. For example, the data-driven tool is fast, since the process simple and the model trained in advance. While the 2D guided tool, such as ACDC, is slower, since the process is complex and included several procedures including 2D segmentation, 3D reconstruction, assets matching, and pose optimization. Another reason is that we add physical optimization in the executor to ensure the physical plausibility in the geometric level, while previous work only consider the bounding box level. The third reason is because we take several steps to develop a scene rather than single step.
The average time for single iteration 8.6 minutes, and the average time for generating complete scene 64 (min:35, max:130) minutes. Simpler and smaller room need less generation time. Here choices reduce the time cost:
reduce the usage frequency of the physical optimization
remove some time-consuming tools, such digital cousins. Although the results will be influenced, it will not cost too much drop.
reduce the iteration number.
C.5 API Cost
The average iteration is about 7. The average API cost is about 0.5 dollar scene. The relatively higher cost come from well-designed agentic framework, making the great results both physical and visual & semantic metrics is irreplaceable by other methods. More diffusion steps or longer Infinigen procedure will not change the difference.
Moreover, here are some choices to reduce the cost by:
simpler requirements and smaller room size.
reduce the usage frequency of the physical optimization.
remove some time-consuming tools replace with other effective tools.
reduce the iteration number.
Note reducing iteration number will not affect the final result too much since the scene achieves high score quickly in the first few iterations. The rest iterations aim to fix corner cases and improve the
A10

---

score slightly. For example, the GPT score for bedroom is 8.6, 9.3, 7.5, when we limited the steps 3, still better than the baseline methods.
D
Limitation
D.1 Unrealistic Scaling & Placement for Functional Items
small ratio objects may look unreal in the scale and placement. This problem comes from the non-standard open-vocabulary dataset in two points. On one side, the object in open-vocabulary dataset such as Objaverse has no standard size, for example, a toy can be 10 meters in length. Thus it is necessary to resize those assets to eliminate the gap and fit the realness. On the other side, the front direction the asset is not provided in the dataset, thus we utilize VLM to predict the front direction, which we find with high success rate, but it still failed in front direction of the asset can not be defined uniformly, such as the elliptical bike, our expected front direction is mistaken as the side direction, leading to unreasonable orientation and size. How to balance the standardization and generality for the open-vocabulary dataset is an unsolved problem, but it still provides a feasible and promising view for general scene synthesis.
since sometimes the
D.2 Validity of LLM's Judgement
To evaluate the generation system, human study stands for a fair choice, but it is very expensive to scale. The LLM provides cheaper, faster and more stable (as discussed in Sec. B.5) choice to evaluate. Taking LLM for both generation and validation seems conflict, while here we consider it as acceptable, since the LLM score is easy acquire during generation and does not result in overfitting. Previous work GPTEva13D [36] have also proved that VLM can serve as Human-A -Aigand Evaluator for Text-to-3D Generation. SceneCraft [37] also utilizes VLM critic and revise the scene with Blender script.
We acknowledge that LLMs are imperfect proxies for human judgment. Alternative scaffolds (e.g.. rule-based metrics, heuristic checks, or hybrid human-Al pipelines)) exist but offer no clear advantage in alignment scalability. In the absence of better accessible evaluators, treat LLM feedback as a practical interim solution. And we remain hopeful that future research will yield more reliable, scalable, and human-aligned evaluation paradigms.
Based on the current situation, we emphasize that, given an accessible LLM model to guide the generation system, the key point is how to utilize the guidance and find a good way to improve the scene quality. Using GPT's judgments during iteration also demonstrates the unique ability our reflective agentic framework. Other baseline methods with fixed pipelines and limited tools lack the ability adjust results based on evaluation scores or LLM feedhack, while ours makes it possible to iteratively refine scenes through feedback-drin driven planning with modular tools.
A11

---

Table A8: Metadata of Initializer: Real2Sim MetaScene.
Initializer: Real2Sim MetaScene: Metadata
Description
Load the most related scene from the Real2Sim indoor scene dataset MetaScenes as the basic scene. Ideal for generating foundational layouts for common room types.
Supported Room Types: living room, dining room, bedroom, bathroom, kitchen, hotel, office, laundry room, and classroom.
Use Case Create foundational layout.
Strengths Provides ready-made layout based on real-world data. Rich of details. Weaknesses Fixed layout, need to modify with other methods to meet user demand. Input Roomtype, Ideas to init the scene.
Table A9: Metadata of Initializer: Model PhyScene.
Initializer: Model PhyScene: Metadata
Description
Using PhyScene, neural network, to generate scene the basic scene. The model is trained on the 3D Front indoor dataset.
Supported Room Types Living room, bedroom, and dining room.
Use Case Create a foundational layout.
Strengths Room clean and tidy. Assets good quality.
Weaknesses Fixed layout with less details.
Input Roomtype, Ideas to init the scene.
Table A10: Metadata of Initializer: LLM GPT.
Initializer: LLM GPT: Metadata
Description Using GPT to generate the foundamental scene.
Supported Room Types
any room type.
Use Case Create an accurate and foundational layout.
Strengths Align well with user demand. More details. Highly versatile and capable of generating scenes for any room type and complex user requirement. Flexible with respect to room design and customization.
Weaknesses Less spatial rationality. May not be real data-driven and Real2Sim methods. Input Roomtype, Ideas to init the scene.
Table All: Metadata of Implementer: 2D Guided ACDC.
Implementer: 2D Guided ACDC: Metadata
Description Using image generation and 3D reconstruction to add additional objects into the
current scene.
Use Case Add group of small objects on the top of empty and large furniture, such as table, cabinet, and desk when there nothing on its top. Strengths Real. Excellent for adding group of objects with inter-relations on the top of large furniture.(e.g.. enriching tabletop), such as adding desk and ollatespoonn,fodl set on the dining table. Accurate in rotation.
set on
Weaknesses Can not add objects on the wall, ground, or ceiling. Can not add objects inside a container, such as objects in the shelf. Can not add objects when there is already something the top.
Input Ideas to add objects.
A12

---

Table A12: Metadata of Implementer: LLM GPT.
Implementer: Implementer: LLM GPT: Metadata
Description
Using GPT to add additional objects into the current scene.
Use Case
Add large objects in the current scene.
Use Case 2 Add 1-2 small objects on the top of small supporting furniture, such as nightstand and cabinet, when there is enough space. (e.g., add cup on the nightstand).
Use Case 3 Add several small objects on the top of large supporting furniture, such as dining table and desk, when there enough space, (e.g., add daily tableware on the dining table), Use Case 4 Add several small objects inside the large furniture. (e.g., add books in the shelf). Use Case Add functional objects or decorations on the wall. (e.g., add painting, mirror, and TV on the wall).
Strengths The location is accurate. Can add objects inside a container, such as objects in the shelf.
Weaknesses
The rotation of asset is not always accurate. Relation between small objects not clear. Can not modify objects in the current scene. Can not add objects on the ceiling.
Input Ideas to add objects.
Table A13: Metadata of Refiner: LLM Remove Object.
Refiner: LLM Remove Object: Metadata
Description
Remove objects with GPT. Works with all room types.
Use Case Remove redundant and unnecessary objects when the scene is crowded when there are too many objects. (e.g., eliminate a table in the corner) Use Case Remove objects that does not belongs this roomtype. (e.g., eliminate the bed in the dining room)
Remove objects when the collision/outside problem has not been solved for several attempts by other tools. (e.g., eliminate the object outside the room) Use Case Remove small objects (usually with collision or outside the supporting surface) when their supporter or container has no enough space to support them. (c.g., eliminate some small objects when the nightstand overloaded)
Use Case 3
Strengths Excels at removing specific objects. Can solve collison and crowded problems directly.
Weaknesses Can not add objects replace objects. You must use this method carefully to avoid mistaken deletion.
Input
Ideas to remove objects.
Table A14: Metadata of Refiner: M.MM&Rll Add Relation.
The relation types introduced in Sec. A.5.
Refiner: LLM& Add Relation: Metadata
between objects the current scene according the layout. Sometimes the relation is encoded in the layout coordinate rather than represented explicitly, making it difficult manage. Explicit relations will make the scene more tidy. Note: Each object can have only one parent object (except for the room). Do not add relation between small objects.
Description Add explicit relation
The optional relations between objects are ieolation_ttpest
Use Case Add explicit relation between large objects, according to the layout, make the scene better-organized.
Use Case Add new relation between large objects, make the scene cegter-organized.
Use Case 3 Add againts wall relation to large objects, make the objects stand against wall. Use Case Add floating small objects on/in a large object.
Strengths Can add relation between objects, make the scene tidy and well-organized quickly. Weaknesses Can not fix the layout problem, such as placing the object into the right place accurately.
Input Ideas to add relation.
A13

---

Table A15: Metadata of Refiner: VLM Update Rotation.
Refiner: VLM Update Rotation: Metadata
Description
Adjust object rotations with GPT to optimize room layout.
Use Case Fix incorrect object orientations, such as bed facing the wall or a chair turned away from desk.
Use Case 2 Improve spatial organization by aligning objects more naturally with the room structure usage context (e.g., rotate sofa face TV chair face a table). Strengths Helps improve the visual and functional coherence room. Can automatically identify misaligned items and suggest better orientations based on typical room usage. Weaknesses Does not move, add, or remove objects. Only focus on rotation. Input Ideas to update rotation.
Table A16: Metadata of Refiner: LLM
Update Size.
Refiner: LLM Update Size: Metadata
Description Modify Object Sizes with GPT. Best suited for significant size adjustments rather
than minor refinements.
Use Case Resizing objects with abnormal proportions (e.g., an object on a table that is over meter tall).
Use Case 2 Scaling objects to meet functional requirements (e.g., enlarging a table when a larger one is needed).
Strengths Effective at adjusting specific object sizes. Weaknesses Cannot modify overall room dimensions. Should only be used when necessary due to potential scene inconsistencies.
Input Ideas to update size.
Table A17: Metadata of Refiner: LLM Update Layout.
Refiner: LLM - Update Layout: Metadata
Description Modify layout with GPT.
Use Case Adjust objects' placement when the objects not well-placed
Use Case Change objects' scale when the size does not match the requirement.
Strengths Excels modifying specific objects. This method is not recommended for slight layout adjustments. is better suited for major changes when necessary.
Weaknesses Can not solve all the problem when the room is crowded. Poor in modify rotation. May lack precision and occasionally overlook details. Can not obey the current relation, such as move object away from the wall when the object is against wall. Can not add objects. Input Ideas to update layout.
A14


Reviews:
[{"review_id": "dvXd1dlPaC", "reviewer": "Reviewer_kZsN", "summary": "This submission introduces SCENEWEAVER, a reflective agentic framework that aims to unify scene synthesis paradigms through tool-based, iterative refinement. The framework employs an LLM-based planner to iteratively build a 3D scene via a suite of extensible scene generation tools. It performs self-evaluation of physical plausibility, visual realism, and semantic alignment with user input. SCENEWEAVER brings the reason-act-reflect paradigm to 3D scene synthesis and the authors claim that it is the first agentic framework for this purpose. They perform experiments on open-vocabulary scene synthesis to demonstrate its effectiveness compared to existing methods with respect to visual realism, physical plausibility, and instruction following. They also provide ablation studies to provide mechanistic insight into the proposed framework.", "questions": "As alluded to in the weaknesses section, I believe the manuscript would improve with some attention to the evaluation. Is there a justification for why a higher average number of objects in the scene (#Obj) is better? Could #Obj be used to weight #OB/#CN (as it stands to reason there are more opportunities for errors given a higher object count)?\nI like the inclusion of the human study, especially the pair-wise comparison. It would be powerful to add some statistical weight to this aspect of the evaluation by increasing the sample size and demonstrating additional statistics like inter-annotator agreement.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "The strength of this submission lies in the integration of the synthesis tools (both generation-focused and evaluation-focused). Because these tools are seamlessly integrated in the framework, the ReAct-style reasoning and planning pipeline for 3D scenes becomes tractable. And because LLMs are now driving the synthesis, open-vocabulary becomes a strength of the generation process. While self-reflection (or at least consistency checks) in 3D is not wholly novel, connecting these established paradigms in a meaningful way through tool use is compelling. That being said, without the ability to review the code, the extensibility of the framework remains unclear. \nThe two weaknesses of this submission both lie in the evaluation. First, leaning on VLM-as-a-judge for visual and semantic evaluation seems problematic as the same model is being used as both agent and judge (this is setting aside that VLMs are known to struggle with spatial reasoning). While it has become common practice, I would like to see stronger citations than 2 preprints as a basis for adopting it.   Second, it is unclear to me that the chosen metrics for physical evaluation are desirable (specifically the number of objects in a scene). Furthermore, claiming best performance when there is a tie (as seems to be the case often with out-of-boundary objects (#OB), and collided object pairs (#CN)) seems incorrect.", "was_revised": true, "final_justification": "The authors have addressed my concerns and their responses to the other reviewers has been instructive. The proposed revisions to the work regarding limitations and the improved user study confirm my original view and understanding of the work. As a result, I maintain my rating.", "rebuttal": "Thank you for the valuable feedback and suggestions.\n### Q1: Explanation of physical metrics\nWe refer to PhyScene to evaluate out-of-boundary objects `#OB` and collided object pairs `#CN`, which indicate the reasonbale layout from two aspects. Although there is a tie between these 2 metrics, we could also achieve the best performance for both of them, indicating the effectiveness of a well-designed agent framework. Meanwhile, we refer to IDesign and evaluate the number of objects `#Obj`. A higher number of objects means more objects to interact for embodied AI and also leads to higher diversity of the generated scene.\n\n### Q2: Increase sample size and inter-annotator agreement in user study.\n**User Number and Sample Size**\n\nIn the main paper, we following prior work LayoutVLM for the user study, thus we only recruited **5** students. To make the user study more solid, we improve the user number to **20** and increase the evaluated scene number to **200**, each of them is shown  40 samples.  We show them the same instructions used as prompt in the LLM reflector for the participant to evaluate the scene carefully. \nWe statistic the results into the following table, which indicates similar results with the previously shown Table 3 in the main paper.\n\n| Method      | Real.  $\\uparrow$   | Func.   $\\uparrow$  | Lay.   $\\uparrow$   | Comp.    $\\uparrow$ |\n|-------------|-----------|-----------|-----------|-----------|\n| LayoutGPT   | 5.83     | 6.21    | 5.26    | 5.99    |\n| I-Design    | 6.65     | 6.57     | 5.73     | 6.79     |\n| Holodeck    | 6.70     | 7.35     | 6.67     | 7.45     |\n| ours        | 8.80    | 8.85     | 8.55     | 8.98     |\n\n**Score Alignment and Validity of Evaluation**\n\nNote the evaluation results of human-evaluation and LLM-score can not be exactly the same, thus a slight difference between them is a normal phenomenom.\nTo further confirm the stability of these justification method, we check the ailgnment between different users as well as LLM.  We convert the users' and LLM's scores on different scene to rankings and calculate Kendall’s Tau for the 4 metrics:\n\n| Alignment        | Real.   $\\uparrow$  | Func. $\\uparrow$    | Lay.  $\\uparrow$    | Comp. $\\uparrow$    |\n|------------------|---------|-------|-------|-------|\n| User-User     | 0.43\t| 0.42\t|0.45\t|0.40|\n| User-LLM        | 0.46\t| 0.45\t|0.48|\t0.55|\n\nResults show strong user-user agreement and user-LLM agreement on different metrics. And we find that user-LLM has higher alignment score than user-user, which indicates that LLM is more stable than user in evaluation. These alignment results also demonstrate the effectiveness of using LLM for justification in Table 2/3 of the main paper.\n\n**Diversity**\nThe diversity of our method is high, due to the extendable tool cards and generation ability of foundation model. We implement the user study to evaluate the diversity of scenes. \n\nSpecifically, we increase the generated scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system.\nWe ask the participants to choose:\n\n- Which method do you prefer?\n- Which method has greater diversity?\n\n| Method   |   w/ LayoutGPT | w/ I-Design | w/ Holodeck |\n|--|--|--|--|\n| Preference  | 94.30% | 91.40%  | 87.40%  |\n| Diversity | 95.60% | 98.90%  | 90.00%     |\n\nResults shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.\n\n### Q3: stronger citations for VLM judgement\nThanks for this suggestion.  We have checked the preprinted citations in our submission, and the **LayoutVLM** and **FirePlace** has been accepted to CVPR 2025, which utilize VLMs for 3D spatial reasoning. Additionally, **GPTEval3D**[1](accepted by CVPR 2024) proves that VLM can serve as a Human-Aligned Evaluator for Text-to-3D Generation. **SceneCraft**[2] (accepted by ICML 2024) also utilizes VLM to critic and revise the scene with Blender script. We will update the citation in the final version.\n\n\n\n[1] Tong Wu, Guandao Yang, Zhibing Li, Kai Zhang, Ziwei Liu, Leonidas Guibas, Dahua Lin, and Gordon Wetzstein. Gpt-4v(ision) is a human-aligned evaluator for text-to-3d generation. In CVPR, 2024.\n\n[2] Hu Z, Iscen A, Jain A, et al. Scenecraft: An llm agent for synthesizing 3d scenes as blender code. ICML, 2024."}, {"review_id": "3KvQzXE8eq", "reviewer": "Reviewer_Er4e", "summary": "This paper introduces a system, Sceneweaver, for generating simulable single room scenes from open language prompts with \"reason -> act -> reflect\" strategy. The system primarily leverages a central planning MLLM with mixtures of LLM,MLLM,VLM and some model-based algorithms as \"tools\" to initialize and iteratively refine a scene. These tools scaffold the iterative process and provide clear direction to the central planner about what to refine and how to do so. The tools are broken into \"initializers\", \"microscene implementers\", and \"detail refiners\" with a few of each type implemented, though more could be added. The authors validate their results against SotA systems with quantitative analysis and small scale user study. They also demonstrate a potential robotics simulation application by loading the generated scenes into a physics simulator for teleoperated manipulation.", "questions": "- We know there isn't contact between objects from the metrics, but what about excess space? If simulation is run on the scenes, how much shifting occurs in the states? Are object placements stable?\n- How do we iterate on a system like this? Since the system relies so heavily on the mixture of tools and their implementations, what metrics would we use to evaluate its improvement when we add or remove individual tools?\n  - Can we evaluate the impact/effectiveness of a single tool in isolation?\n- \"the secret sauce\" - looks like we're missing details about the implementation of the tools themselves. Beyond the tool cards and scattered discussion we're lacking concreteness on these details.\n  - e.g. how exactly is the \n- We see \"functional diversity\" as a focus for this project, but we don't see any analysis of diversity specifically mentioned. Is this feasible to add for a small prompt domain? How much visual and content diversity can one achieve with a fairly precise prompt?", "limitations": "Primary limitation claimed by the authors is the generation time and complexity.\n\nLikely, other limitations related more to the specific mixture of tool ingredients present in this iteration rather than a fundamental system limitation given that tools can be expanded. However, they could mention: \n- limited complexity of relationships - mouse+keyboard+monitor is a good start, but real robotics tasks will require much more and \"well aligned\" content\n- hallucinations of realism - e.g. \"squashing\" of functional items like exercise equipment or blocking of articulable items like drawers.\n- clearly no structural elements are considered (windows, doors, stairs) and rooms are always rectangular\n- functional access does not seem to be validated: can an agent reach the furniture or objects to use them?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "I like the approach. Iterative language model planning with tool use is a clear strategy and the authors executed on this approach to improve on SotA metrics. I do think some more details could be included and more limitations could be addressed, particularly on the diversity of results and anti-physical content. Also, if robotics is a primary target application it would be good to try the system in task contexts to validate that it generalizes beyond simple open prompts. A user study was nice to see and results were clear, but more than 5 participants would be nice. Overall I'm recommending a weak accept but would consider stronger support with some more details added.\n\nPros:\n- open-language input\n- focus on physically interactive scenes targets robotics simulation well\n- closed loop iterative refinement  vs. monolithic\n- pre-defined \"tools\" inject the hand-designed or model-based organized elements which MLLMs can leverage more openly\n    - the \"secret sauce\" is the combination and implementation of these tools\n- human study to validate the results\n- Captures hierarchical support relationships between objects\n- extensible with more tools\n- supports multiple input modalities (2D guided vs. data-driven)\n- as models improve, so should results\n- because tools have language feedback, it can be understandable compared to quantitative optimization algorithms (e.g. based on statistical priors)\n- object relationships are considered explicitly (e.g. chairs facing the desk)\n\nCons:\n- single room scale\n- lots of prompt engineering\n- unrealistic scaling/placement for functional items (e.g. the gym layout with vertically stacked treadmills and squashed elliptical bikes.\n    - At the end of the day, function dictates parameter limits. You can't scale an elliptical machine and a chair must be large enough to fit a person. These hallucinations limit the effectiveness of the final scenes.\n- Large generation times (up to hours) limits utility in practice\n- minor (out of scope): no articulations or considerations - e.g. drawers and drawers need to open\n- 5 users study with 20 scenes each seems a bit lightweight\n- No task-based specifications (yet?) - interesting to see that robotics simulation was the target here, but task doesn't manifest in examples. I suppose a prompt could include the task like \"generate a kitchen scene with dishes next to the sink for dish washing\", but it would be interesting to see that demonstrated if so.", "was_revised": true, "final_justification": "My understanding of the work and my opinion of the strengths and weaknesses have not changed, nor has my rating which was provided under the assumption of my understanding. Overall I am satisfied with the additional information provided by the authors to answer my questions and confirm where my criticisms should be mentioned as additional limitations and avenues for future work.  I feel the final version of the exposition will be improved by the increased transparency and additional details.", "rebuttal": "Thank you for the valuable feedback and suggestions.\n### Q1: More participants in user study.\n\nWe enhanced the study's robustness by scaling to 20 participants evaluating 200 scenes (40 samples each), using identical LLM reflector instructions to ensure consistent evaluation criteria. The results remain consistent with Table 3's findings.\n\n| Method | Real.  $\\uparrow$   | Func.   $\\uparrow$  | Lay.   $\\uparrow$   | Comp.    $\\uparrow$ |\n|-|-|-|-|-|\n| LayoutGPT| 5.83 | 6.21 | 5.26| 5.99|\n| I-Design| 6.6 | 6.57 | 5.73     | 6.79   |\n| Holodeck| 6.70| 7.35 | 6.67 | 7.45  |\n| Ours  | 8.80| 8.85 | 8.55| 8.98 |\n\n### Q2: accessibility of the furniture\n\nThis is a good suggestion.  We think it is a good point to add this aspect into a new physical metric. Here we refer to PhyScene and SceneEval and add additional physical metrics “accessibility”.  \nDue to the time limits, here we randomly choose 3 bedrooms of each method and check the percentage of large furniture (such as bed, wardrobe, and nightstand) that can be reached in the front.\n\n| Method| Accessibility |\n|-|-|\n| LayoutGPT | 77% |\n| IDesign   | 80%|\n| Holodeck  | 82% |\n| ours  | 82%|\n\nOur methods shows high accessibility since the agent is able to check the walkable area. For example, the step 4 in Figure 1 shows the agent has recognized the crowding area and fix it. Note the Holodeck shows relatively high accessibility. That’s because its room has larger size than other methods and objects are closed to the wall.\n\n### Q3: Release more details and limitations such as diversity of results and anti-physical content. \n\n**Diversity**\nThe diversity of our method is high, due to the extendable tool cards and generation ability of foundation model. We implement the user study to evaluate the diversity of scenes. \n\nSpecifically, we increase the generated scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system.\nWe ask the participants to choose:\n\n- Which method do you prefer?\n- Which method has greater diversity?\n\n| Method   |   w/ LayoutGPT | w/ I-Design | w/ Holodeck |\n|-|-|-|-|\n| Preference  | 94.30% | 91.40%  | 87.40%|\n| Diversity | 95.60% | 98.90%  | 90.00%|\n\nResults shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.\n\n**More Limitations**\n\n- Asset retrieval may return poor results due to non-standard open-vocabulary formats and language embedding corner cases.\n- For assets not generated by Infinigen's procedures, supporting surfaces are estimated from geometry. If the geometry is irregular, this can result in tilted surfaces and unusual placement of child objects.\n- The previous tool produced sparse layouts inside containers, unable to densely arrange objects because it couldn't accurately detect supporter shapes. To address this, we extended the tool to enable denser placements, such as adding 30 well-arranged books on a shelf.\n\n### Q4: More practical metrics：Shift in simulation\nThanks for your suggestion.   To assess object stability insimulation, we measured:\n- Shift thresholds: percentage of objects moving >0.1m or >0.01m in 3s\n- Average displacement: Mean shift distance (meters)\n\n| Method |  > 0.1m  | > 0.01m | Avg Shift  |\n|-|-|-|--|\n| ATISS | 35.40% | 51.40%  | 0.356 |\n| DiffuScene  | 26.20%  | 39.30%  | 0.190  |\n| Physcene| 9.70%  | 19.60%  | 0.069  |\n| LayoutGP | 39.20% | 52.80%  | 0.477   |\n| IDesign | 5.00% | 11.50% | 0.041 |\n| Holodeck | 17.60% | 42.50%  | 0.113  |\n| Ours | 1.00%| 10.37% | 0.011 |\n\nThe lowest shift of our mthods verify our great performance on the original physical metrics #OB and #CN. Obove all, our generated scene remains the most stable in simulation although with the largest object number.\n\n### Q5：generation with more complex prompts for robot task\n\nThis is a good point. The robotic task context can be achieved by adding specific description prompt design. Here we show some generated samples of complex prompts in Figure A3. Our method is capable of following complex requirements in different context.  It shows the controllability of objects’ category (basket, washtub, etc.), number (”A laundromat with 10 machines”), position (”a car in the center”), and precise components (decoration on the wall, small objects).\n\n### Q6: Evaluate the impact of a single tool\n\nTable 5 shows that adding different types of tools (Initializer, Refiner, Implementor) greatly improves results. To assess the effect of a single tool, results can be compared with and without that tool using identical prompts, which requires multiple generations for reliable statistics. Due to time constraints, we do not provide precise quantitative results here. However, we observed that without the “Update Rotation” tool, the model is less sensitive to invalid rotations, reducing the “layout” score. Meanwhile, the “Add Crowd” tool lets supporters be densely filled with child objects (e.g., a shelf packed with books), which improves scene realism. If there is multiple tools in similar function, adding/removing a single tool will not make big difference. And if the tool has negative effect, the agent will recognize it during iterations and avoid using the tool.\n\n### Q7: Room structure & single room scale\n\nWe remove windows and doors to simplify the generation process, though our method can handle such structures and update the scene accordingly. Due to the format limitation of rebuttal, we can show some examples in the final version.\n\nFor room scale, we focus on single rooms to improve scene quality as prompted by the user. The process can be repeated to handle multiple rooms. We could also generate multi-room scenes at once with some coordinate transformation.\n\n### Q8: time limits\n\nThe average time for a single iteration is 8.6 minutes, and the average time for generating a complete scene is 64 (min:35, max:130) minutes. Simpler and smaller room need less generation time. Here are some choices to reduce the time cost:\n- reduce the usage frequency of the physical optimization \n- remove some time-consuming tools, such as digital cousins. Although the results will be influenced, it will not cost too much drop.\n- reduce the iteration number.\n\n### Q9: unrealistic scaling/placement for functional items\n\nThat is a good point. The problems comes from the non-standard open-vocabulary dataset in two points. On one side, the object in open-vocabulary dataset such as Objaverse has no standard size, for example, a toy can be 10 meters in length. Thus resize is neccessary for those assets to eliminate the gap and fit the realness. On the other side, the front direction of the asset is not provided in the dataset, thus we utilize VLM to predict the front direction, which we find with high success rate, but it still failed in some cases since sometimes the front direction of the asset can not be defined uniformly, such as the elliptical bike, our expected front direction  is mistaken as the side direction, leading to unreasonable orientation and size. \nHow to balance the standardization and generality for the open-vocabulary dataset is an unsolved problem, but it still provide a feasible and promising view for general scene synthesis.\n\n### Q10: Articulations and squashing.\nThat’s a good point. We will pay attention to the articulation and squashing in the future work.\n\n### Q11: limited complexity of relationships\nThe relationship is taken from the Infinigen with 3 types of obj-room relations and 7 types of obj-obj relations. This covers most of the common relationship. For more \"well aligned\" content, the code is also easily extendable for more complex and detailed relation.\n\n### Q12: Implementation of the  tool & executor\nWe list the implementation of each tool. \n\n | Tool Name | Role | Use of Existing Works  | Our Contribution   |\n|--|-|--|--|\n| Init MetaScenes | Init scene with Real2Sim dataset  | MetaScenes| Choose data + convert format|\n | Init PhyScene | Init scene with pretrained model  | PhyScene/DiffuScene/ATISS   | Choose data + convert format  |\n | Init GPT | Init scene with LLM    | LLM  | Prompt engineeri   |\n| Add ACDC| Add tabletop objects visually| Stable Diffusion + ACDC| Significant changes on digital cousin|\n | Add GPT | Add objects with LLM  | VLM | Prompt engineering|\n| Add Crowd  | Add crowded layout   | LLM+Infinigen| Utilize Infinigen rules + design module |\n| Refiner | Remove Object| Remove invalid objects| LLm   | Prompt engineering |\n | Add Relation | Add relations to objects | VLM+ Infinigen  | Utilize Infinigen relations + prompt engineering |\n | Update Rotation  | Fix rotation problems   | VLM | Prompt engineering|\n| Update Size | Rescale objects   | LLM  | Prompt engineering |\n | Update Layout| Update improper layouts  | VLM | Prompt engineering |\n\nDuring the planning process, the agent is provided with brief description of each tool, shown in A3-A13 in supplementary material and use the function calling to choose a single tool for each step. Then it will run the tool itself and go through the executor to update the scene.\n\nNote we not only contribute on the tool itself, but also modify the infinigen to :\n- update iteratively\n- interactive in blender in realtime with socker\n- fit each tools rather than generate scene procedurelly.\n- Physical optimization\n- Add 3D marks and 2D top-down rendering\n- …\n\n### Q13: prompt engineering\n\nWe appreciate your insightful observation regarding the prompt engineering efforts in our work. We follow the structure of **Octotools** and design prompts for planner, tool cards,  executor, and reflector, which makes the framework  accurate, stable, and easily extensible. Other baseline methods (IDesign, LayoutVLM, Holodeck) also design several prompt for each step, which is a common choice for using the LLM and agent for generation task. \n\n### Q14: Typos error\nThanks for pointing out these problems. We will fix the typo error in the final version."}, {"review_id": "JN8z1PgUiz", "reviewer": "Reviewer_DJFe", "summary": "The paper presents SceneWeaver, an LLM-based system which integrates many asset libraries and generation tools to perform open-vocabulary language-conditioned generation of indoor 3D environments. SceneWeaver uses multimodal language models to iteratively suggest improvements (\"reflection\") and choose tools to use to add assets, add physical constraints, or correct errors detected by the LLM.", "questions": "Are the existing qualitative results random and non-selected? What is the variance in visual quality if I evaluated random seeds 0...5 with the exact same bedroom prompt?\n\nWhat is the technical contribution of the tool implementations and standardized tool cards? How do the tool cards expand on a normal LLM-tool-use setup e.g. describing the tools to the openai api? The presence/non-presence of tools is ablated, but _standardized and extensibility_ is listed as a contribution (L74). \n\nWhat parts of the \"executor\" (L202) are built by the authors vs taken from existing tools?\n\nI would appreciate any discussion or further justification of Tab2/3/6 quantitative results (see above). In my view, the LLM-as-judge results should be removed from the paper.\n\nWhat is the mean/min/max SceneWeaver cost in terms of CPU-hours, GPU-hours and API credits? What are the costs for some of the examples scenes in the main paper? What are the realism metrics if SceneWeaver is restricted to the runtime of the baselines? How do we know that the cost-realism tradeoff of SceneWeaver is favourable for downstream tasks (robotics/games) compared to baselines?\n\nI believe the work is promising and would argue for acceptance if the authors address my concerns on evaluation and cost.", "limitations": "Runtime is listed as a limitation (which is good to report) but requires further details discussion (see above).\n\nNo \"potential negative impacts\" are mentioned but I do not see this as significant.", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Qualitative results:\n- strength: the qualitative results provided are compelling, especially offices and gyms which are challenging (due to objects not attached to walls, e.g. grid of desks or treadmills) and are not supported in prior tools e.g. ProcThor, Infinigen \n- weakness: lacking a larger controlled qualitative sample. ATISS and Infinigen both provide 30+ random qualitative samples in their supplement. The paper would be stronger if it included such a sample, ideally seeds=0,1,2 for each language prompt (so we could have seen the variance due to random seed) and with outputs for other methods for the same prompt or scene category. \n\nTool use:\n- strength: the work integrates an impressive variety of object and scene synthesis tools, which I believe takes nontrivial engineering effort\n- weakness: it is unclear in the main paper which tools use what underlying woreks. paragraph L199 should be significantly expanded in the main paper to make the technical implementation of these tools clear. \n\t- The work uses assets from Objaverse, 3D-Future, Infinigen \"depending on the tool\" (L204) but doesnt state how these choices are made (even in A.3). L75 states tools are used \"based on their respective strengths\" - how does the system know or encode these strengths?\n\t- It is unclear how object resizing works - are the assets scaled up/down (unrealistically?) or does it use infinigen parameters? \n\t- A.3. specifies the arrangement constraint system, but it extremely closely resembles Infinigen's solve_state.json. \n\t- The paper main paper could specify this all carefully (ideally a table with columns \"tool name\", \"role\", \"our contribution\", \"use of existing works\") and expand paragraph L199. \n\nMinor issue: Table 2/3's `#OBJ, #OB and #CN` metrics are gameable\n- These metrics are easily saturatable (many works score 0.0 error), and in theory can be satisfied by a weak baseline (e.g. a shape-packing algorithm that places objects without intersection but with no regard for realism or even gravity). \n- `#OB` and `#CN` are necessary but not sufficient for use in robotics sims. e.g. objects could be floating or topple over in the sim, but wouldnt violate these metrics. Some objects in the demo video would topple in this way.\n- A better metric would be \"what is the sum of total pose changes when we initialize a simulation and let objects settle?\". This would subsume `#OB` and `#CN` (since intersections cause large movements), but would also measure other bad phenomena (object topples over, object drops to the floor). https://arxiv.org/abs/2405.20510v1 and similar use metrics along these lines I believe. \n\nTable 2/3's use of GPT-4 as a judge of \"Realism\", \"Functionality\" and \"Layout\" is invalid for SceneWeaver, (even despite that \"LLMs as judge\" is now a common paradigm for other work). \n-  The SceneWeaver reflection system uses GPT-4 (the exact judge model, mentioned on L178) to actively iterate against the exact \"real/func/layout\" validation scores (confirmed in Appendix Table A2) . If the method accesses the validation scores during generation, then in my view they are invalid as validation scores.  This is similar to reporting the train loss of a deep neural network, or training on a validation set. If the comparison setup of Tab2/3 says this is fair game, ATISS should be allowed to generate 1000 samples and return whichever GPT-4 says is best, before then being judged by GPT-4.\n- A premise of the work is that LLMs need significant scaffolding (e.g. SceneWeaver) in order to correct deficiencies in spatial reasoning. However, the work then uses GPT-4 with minimal scaffolding as quantitative evaluation. \n- A more minor issue: Using natural language prompts as the definition of \"functionality\" is imprecise. Functionality could have a precise geometric definition, e.g. what affordances are preserved (% of sofas/seating that can be accessed from the front, or % of percent of ovens/cabinets that have space to open without obstruction). These are directly important for downstream tasks. I recognize this is a minor and very stringent criticism however, as most prior works do not evaluate this.\n\nThe authors do provide a human user study of 5 participants, which partially addresses my concerns in Tab2/3. In my view, this needs to serve as the main evaluation of the system. Therefore, it would benefit from more participants and a stronger study design to reduce variance. Specifically the user study would ideally:\n- (1) clearly define what evaluators should look for in terms of \"real\", \"functional\" and \"layout\" (currently there are no instructions to participants what these mean - see appendix A7)\n- (2) provide an evaluation of layout which specifically says to ignore asset / rendering quality (similar to Infinigen-Indoors) \n- (3) show multiple outputs for each generation system, to evaluate which _distribution_ of results is better (evaluating individual samples will be high variance) \nOverall, the human user study shows much more minor gains over Holodeck than the GPT-4-as-judge study, which lends credence to the idea that LLMs-as-judges may not be valid for this paper.\n\nThe work does provides insufficient description and argument on the cost of the system. AppendixC-L76 states it is \"minutes to hours\" but this could mean 24 hours. The work uses GPT-4, so it also has a precise dollar-cost of APIs used per scene. CPU cost, GPU cost, and API cost should each be reported as a mean, min and max, or even reported for individual qualitative results, or means reported in Tab2/3 for each method if available. Along with this, I would expect an argument that the cost is worthwhile for the improved results (e.g. \"+X minutes is acceptable for embodied sims because it can be amortized over Y minutes of sim usage\"). If the cost is very very high, it would be rigorous to also evaluate against baselines on an equal-cost setting (fewer reflection steps for SceneWeaver, or maybe increase DiffuScene denoising steps / run Infinigen with very large annealing steps). \n\nIn summary, I believe the work does produce impressive scene arrangements, but would appreciate controlled and more numerous qualitative results. My major concerns are that (1) Table2/3 metrics should not be presented as primary evidence of the work's usefulness and (2) the work does not report detailed cost, does not weigh cost in any comparisons, and does not argue for the acceptability of its cost.\n\n\"Quality\" Justification: technical soundness - LLM-as-judge is not technically sound, requires additional evaluation (qualitative and quantitative). some claims in Tab1 are potentially excessive (\"large-scale\", \"real\"(ism?), \"accurate\" - all are subjective and may apply to baselines)\n\n\"Clarity\" Justification: good except for description of how underlying works are used. \n\n\"Significance\": potentially very useful as a robotics / sim tool. usefulness depends on cost. \n\n\"Originality\" Justification: LLM \"reflection\" or tool use are not original, and i don't believe their application to scene arrangement is highly original, unless I am misunderstanding details of the \"tool cards\".", "was_revised": true, "final_justification": "Table 2&3's evaluation setup is potentially flawed. The metrics are graded by GPT-4, but SceneWeaver's method also prompts GPT-4 to give iterative feedback on these same metrics during generation. Accessing the validation score in this way is invalid. The authors provided new empirical evidence that this does not hugely bias the results, but the conceptual issue has not been acknowledged, and I believe this creates a bad precedent or may mislead readers.\n\nRelated work makes unclear claims via Tab 1 check/cross marks that previous work is not \"accurate\" or \"physically plausible\", which are not binary or objective. This is not major, but was not acknowledged in rebuttal after being raised twice by me and once by DP6K. \n\nThe authors significantly strengthened their other quantitative results during rebuttal, and overall the qualitative and quantitative results are compelling. I maintain a borderline reject on the basis that the Tab 2/3 issues are potentially serious, but Tab2/3 could instead just be removed.", "rebuttal": "Thank you for the valuable feedback and suggestions.\n### Q1: User Study\n**User Number and Sample Size**\n\nWe enhanced the study's robustness by scaling to **20** participants evaluating 200 scenes (40 samples each), using **identical LLM reflector's instruction of metrics** to ensure consistent evaluation criteria. Similar to Infinigen-Indoors' approach of focusing solely on furniture layout's realness without texture and rendering, our original instructions (shown in Table A2) also emphasize to **evaluate realness by excluding texture and lighting considerations.** This unified evaluation framework, combined with our Objaverse-based pipeline, minimizes platform/asset biases. ）The results remain consistent with Table 3's findings.\n\n| Method| Real.  $\\uparrow$   | Func.   $\\uparrow$  | Lay.   $\\uparrow$   | Comp.    $\\uparrow$ |\n|-|-|-|-|-|\n| LayoutGPT   | 5.83     | 6.21    | 5.26    | 5.99    |\n| I-Design    | 6.65     | 6.57     | 5.73     | 6.79  |\n| Holodeck    | 6.70     | 7.35     | 6.67     | 7.45 |\n| ours | 8.80    | 8.85     | 8.55     | 8.98     |\n\n### Score Alignment and Validity of Evaluation\nGiven that human and LLM evaluations naturally differ slightly, we assessed their consistency by converting scores to rankings and computing Kendall's Tau across all four metrics. This confirms the stability of LLM's justification.\n\n| Alignment | Real.   $\\uparrow$  | Func. $\\uparrow$    | Lay.  $\\uparrow$    | Comp. $\\uparrow$    |\n|--|-|-|-|--|\n| User-User | 0.43| 0.42|0.45|0.40|\n| User-LLM | 0.46| 0.45|0.48|0.55|\n\nResults demonstrate strong agreement between users and LLM across metrics, with higher user-LLM alignment than user-user (τ=0.55 vs 0.40 on Completion metric), confirming LLM's superior evaluation stability. These findings validate our LLM justification approach in Tables 2/3.\n\n### Diversity and Preference\n\nWe increase the scene number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different methods.\nWe ask the participants to choose:\n\n- Which method do you prefer?\n- Which method has greater diversity?\n\n| Method | w/ LayoutGPT | w/ I-Design | w/ Holodeck |\n|-|-|-|-|\n| Preference    | 94.30% | 91.40%  | 87.40% |\n| Diversity | 95.60% | 98.90% | 90.00% |\n\nResults shows our methods has greater diversity while also stand for higher preference, surpassing other methods in a large degree.\n\n### Q2: GPT for both generation and validation is invalid. need more scanffolding.\n\nWe understand your concern. To evaluate the generation system, human study stands for a fair choice, but it is very expensive to scale. The LLM provides a cheaper, faster and more stable (as shown in the user-LLM alignment) choice to evaluate. Taking LLM for both generation and validation seems conflict, while here we consider it as acceptable, since the LLM score is easy to acquire during generation and does not result in overfitting. We could also add other scanffoldings, but they may not provide a better evaluation.  \n \nUsing GPT to filter the results of  ATISS will still returns lower score due to the limitation of the method itself, trapped in data distribution of the training dataset with limited categories and room types. Therefore,  given an accessable LLM model to guide the generation system, we emphasize how to utilize the guidance and find a good way to improve the scene quality.\n\n### Q3: “Functionality” could have a geometric definition such as accessibility and interactivity.\nThis is a good suggestion. The “functionality” you mentioned is related to precise calculation for the physical interactivity, which is hard for LLM to reply. Here we refer to PhyScene and SceneEval and add additional physical metric `accessibility`.  \nDue to the time limits, here we randomly choose 3 bedrooms of each method and check the percentage of large furniture (such as bed, wardrobe, and nightstand) that can be reached in the front.\n\n| Method    | Accessibility |\n|--|--|\n| LayoutGPT | 77%   |\n| IDesign   | 80% |\n| Holodeck  | 82%  |\n| ours  | 82%   |\n\nOur methods shows high accessibility since the agent is able to check the walkable area. For example, the step 4 in Figure 1 shows the agent has recognized the crowding area and rearranged the dining tables to ensure adequate space for movement and clear walkways. Note the Holodeck shows relatively high accessibility. That’s because its room has larger size than other methods and objects are closed to the wall.\n\n### Q4:  Physical Metrics\n**Validity of `#OB` and `#CN`**\n\nHere we do not use shape packing or floating placement for higher physical score. The size of assets is decided by each tool in reasonable process. And each object has as least one relation (contact closely with out collision) with the room (floor/wall) or supporting objects (table/shelf/…), ensuring that nothing is floating in the room. Thus table 2/3's `#Obj`, `#OB` and `#CN` metrics are solid.\n\n**New Metric：Shift in simulation**\n\nThanks for your suggestion.   To assess object stability in simulation, we measured:\n\n- Shift thresholds: percentage of objects moving >0.1m or >0.01m in 3s\n- Average displacement: Mean shift distance (meters)\n\n| Method  | >0.1m |>0.01m | Avg Shift  |\n|-|---|--|---|\n| ATISS   | 35.40% | 51.40%  | 0.356 |\n| DiffuScene  | 26.20%  | 39.30%  | 0.190  |\n| Physcene    | 9.70%  | 19.60%  | 0.069  |\n| LayoutGPT   | 39.20%     | 52.80%  | 0.477   |\n| IDesign  | 5.00%   | 11.50% | 0.041 |\n| Holodeck | 17.60% | 42.50%  | 0.113  |\n| Ours | 1.00%  | 10.37% | 0.011   |\n\nThe lowest shift of our mthods verify our great performance on the original physical metrics `#OB` and `#CN`. Obove all, our generated scene remains the most stable in simulation with the largest object number, which further confirm the great ability of our method’s  physically suitable position.\n\n### Q5: Cost \nThe **average time** for a single iteration is 8.6 minutes, and the average time for generating a complete scene is 64 (min:35, max:130) minutes. The average iteration is about 7. \nThe average **API cost** is about 0.5 dollar per scene. And Holodeck report its cost as 0.2 per scene. IDesign and LayoutGPT does not release the cost. \nSince other baseline does not provide **CPU&GPU cost**, we think the comparison might not be precise. \n\nThe relatively higher cost come from well-designed agentic framework, making the great results on both physical and visual & semantic metrics is irreplaceable by other methods. More diffusion steps or longer infinigen procedure will not change the difference. \n\nMoreover, here are some choices to reduce the cost by:\n- simpler requirements and smaller room size.\n- reduce the usage frequency of the physical optimization. \n- remove some time-consuming tools or replace with other effective tools.\n- reduce the iteration number.  \n\nNote reducing iteration number will not affect the final result too much, since the scene achieves high score quickly in the first few iterations. The rest iterations aim to fix corner cases and improve the score slightly. For example, the GPT score for bedroom is 8.6, 9.3, 7.5, 7 when we limited the steps to 3, still better than the baseline methods.\n\n### Q6: Implementation of the tools\nWe list the implementation of each tool. \n\n| Tool Type   | Tool Name    | Role  | Use of Existing Works  | Our Contribution  |\n|-|-|-|-|-|\n| Initializer | Init MetaScenes | Init scene with Real2Sim dataset  | MetaScenes                  | Choose data + convert format         |\n|   | Init PhyScene   | Init scene with pretrained model  | PhyScene/DiffuScene/ATISS   | Choose data + convert format         |\n|   | Init GPT        | Init scene with LLM    | LLM(GPT)    | Prompt engineering |\n| Implementer | Add ACDC        | Add tabletop objects visually     | Stable Diffusion + ACDC     | Significant changes on digital cousin|\n| | Add GPT  | Add objects with LLM   | VLM | Prompt engineering    |\n|  | LLM+Rule        | Add crowded layout                | LLM+Infinigen  | Utilize Infinigen rules + design module |\n| Refiner | Remove Object    | Remove invalid objects            | LLM(GPT)   | Prompt engineering |\n|  | Add Relation | Add relations to objects  | VLM(GPT) + Infinigen  | Utilize Infinigen relations + prompt engineering |\n|   | Update Rotation  | Fix rotation problems  | VLM(GPT) | Prompt engineering  |\n|  | Update Size      | Rescale objects  | LLM(GPT)  | Prompt engineering   |\n|  | Update Layout    | Update improper layouts | VLM(GPT)  | Prompt engineering   |\n\nDuring the planning process, the agent is provided with brief description of each tool, shown in A3-A13 in supplementary material and use the function calling to choose a single tool for each step. Then it will run the tool itself and go through the executor to update the scene.\n\n### Q7: Implementation of the executor\nNote we not only contribute on the tool itself, but also modify the infinigen to :\n- update iteratively\n- interactive in blender in realtime with socker\n- fit each tools rather than generate scene procedurelly.\n- Physical optimization\n- Add 3D marks and 2D top-down rendering\n- …\n\n\n### Q8: How to choose assets dataset\nIn this project, we choose different asset according to the usage of tool. And you can also choose any of them to fit your own requirements. \n- MetaScenes: For tool using Dataset such as MetaScenes, we employ its assets directly, since each scene contains several assets with delicated mesh and layout information.\n- 3D FUTURE: For tool using Model such as PhyScene/DiffuScene/ATISS, we employ 3D FUTURE, since the model is trained on this dataset.\n- Infinigen: For other tools, we use Infinigen's asset generation code to generate standard assets in common categories, such as bed, sofa, and plate. The asset will be generated in a delicated rule procedure in the scene generation process.\n- Objaverse: For those catrgories that are not supported by Infinigen, such as clock, laptop, and washing machine, we employ open-vocabulary Objaverse dataset."}, {"review_id": "lNY8UrSmDc", "reviewer": "Reviewer_dp6k", "summary": "The paper introduces a language-model-based agentic framework to generate both common and open-vocabulary room types. It dynamically selects tools and iteratively refines the scene based on feedback from language models, and it achieves better results than baselines on both human evaluation and language-model evaluation.", "questions": "- Can the author provide real “iterative” examples where the model rebuilds part of the scene to improve each of the metrics: visual realism (Real.), functionality (Func.), layout correctness (Lay.), and scene completeness? Otherwise, it is hard to assess the significance of the approach.\n- Can the author provide experiments on using the proposed framework to refine scenes generated by baseline methods, so that the scenes under comparison look mostly similar but improved?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The framework unifies a lot of existing scene synthesis methods and improves on them.\n- The paper is easy to follow with a lot of figures to help understand.\n\nWeaknesses\n- My overall impression is that the pipeline can be done through simple optimization without language models. From figure 5, it is hard to see the “iterative” part (maybe except for the move of the table). It seems the pipeline is just about coarse to fine object placement, similar to Infinigen.\n- In figure 4, because these scenes are all different, a naive approach could just be to let the model sample a lot of different scenes and (automatically) cherry-pick one. Because the other approaches do not do cherry-picking, it is easy to outperform them. So it would be best to have more restricted comparisons.\n- The evaluation metric is mostly language-model-based. So it is likely “the blind leading the blind.” Though there are human evaluations, they are limited by the number of participants.\n- In table 1, the criteria might be too strict, for example, Infinigen is classified as not “Real”.\n- The controllability of the method is limited, at least in the experiment. The prompt is only the room type.", "was_revised": true, "final_justification": "Most of my concerns are resolved, therefore I increase my rating from 3 to 4", "rebuttal": "Thank you for the valuable feedback and suggestions.\n### Q1: More participants in human evaluations.\nWe improve the user number to 20 and increase the evaluated scene number to 200, each of them is shown  40 samples.  We show them the same instructions used as prompt in the LLM reflector for the participant to evaluate the scene carefully. \nWe statistic the results into the following table, which indicates similar results with the previously shown Table 3 in the main paper.\n\n| Method      | Real.  $\\uparrow$   | Func.   $\\uparrow$  | Lay.   $\\uparrow$   | Comp.    $\\uparrow$ |\n|-------------|-----------|-----------|-----------|-----------|\n| LayoutGPT   | 5.83     | 6.21    | 5.26    | 5.99    |\n| I-Design    | 6.65     | 6.57     | 5.73     | 6.79     |\n| Holodeck    | 6.70     | 7.35     | 6.67     | 7.45     |\n| ours        | 8.80    | 8.85     | 8.55     | 8.98     |\n\n### Q2: GPT for both generation and validation \nWe understand your concern. To evaluate the generation system, human study stands for a fair choice, but it is very expensive to scale. The LLM provides a cheaper, faster and more stable (as shown in the user-LLM alignment) choice to evaluate. Taking LLM for both generation and validation seems conflict, while here we consider it as acceptable, since the LLM score is easy to acquire during generation and does not result in overfitting. We could also add other scanffoldings, but they may not provide a better evaluation. We can also apply GPT to filter the results of other methods such as ATISS, but they will still returns lower score due to the limitation of the method itself. For example, ATISS will be limited to the data distribution of the training dataset, trapped in limited categories of large furniture and simple layout of only 3 room types. Therefore, we emphasize that, given an accessable LLM model to guide the generation system, the key point is how to utilize the guidance and find a good way to improve the scene quality.\n\n### Q3: More controllable prompt \nWe have shown some samples of complex prompts in Figure A2 in the supplementary material. It shows the controllability of objects’ category (basket, washtub, etc.), number (”A laundromat with 10 machines”), position (”a car in the center”), and precise components (decoration on the wall, small objects).\nWe will release more results in the final version and web page, since the rebuttal can not show additional images and external links.\n\n### Q4: Use SceneWeaver to refine scenes generated by baseline methods\nThat is a good idea. Due to time constraint in rebuttal, here we refine scenes generated by two baseline methods, PhyScene and LayoutGPT. (IDesign and Holodeck will take longer time in format conversion.) \n\n| Method      | #Obj | Realism | Functionality | Layout | Completion |\n|-------------|------|---------|---------------|--------|-------------|\n| PhyScene    | 5    | 7       | 8             | 6      | 5           |\n| +ours   | 11   | 9   | 10        | 8  | 9       |\n| LayoutGPT   | 6    | 7       | 8             | 6      | 5           |\n| +ours   | 20   | 9   | 9         | 8  | 10      | \n\nResults shows our method could greatly improve the baseline method.\nActually the baseline method serves as a initializer tool in this setting, and we can convert each baseline method to a new tool and merge them into the extensible tool cards.\n\n### Q5: “Iterative” examples to improve each of the metrics\nThe overall is self-adaptive rather than coarse to fine object placement. The agent in our pipeline will find the biggest problem in each iteration according to the reflection score and choose related tool to solve the problem. For example, the teaser serves as an obvious example. \nHere we show the metric score after executing each step:\n| Step | Tool         | Realism | Functionality | Layout | Completion |\n|------|----------------|---------|---------------|--------|------------|\n| 1    | Initializer       | 6       | 6             | 5      | **4**      |\n| 2    | Implementer        | 7       | 6             | **5**      | 6          |\n| 3    | Modifier  | 8       | 7             |**6**    | 6          |\n| 4    | Modifier         | 8       | 7             | 8      | **6**          |\n| 5    | Implementer        | 8       | 7             | 8      | 8          |\n\nThe agent will choose the lowest score to improve. Specifically, the step 1 has lowest score in completion, which is 4. Then in the step 2, the agent chooses the implementer to add objects in the shelf and improve the completion score from 4 to 6. Now the lowest score is the layout, which is 5, owing to the illogical location of bathroom sink. So in step 3, it uses modifier to remove the bathroom sink, improving the layout score from 5 to 6. So now the lowest score consist of both layout and completion. In step 4,  the agent focuses on the crowding area and decide to use modifier to rearrange the dining tables to ensure adequate space for movement and clear walkways. Then the score of layout becomes 8. The lowest score come to the completion again. So the agent choose implementer again to add objects on each table to improve the completion score. \n\nYou can see the realism and  functionality will also be impoved while we focus on optimizing other metrics. And the agent can fix the problem caused by previous steps, such as removing the bathroom sink generated in previous step. And it can utilize the same tool for multiple times to improve the quality untill it fit the demand.\n\n### Q6: Sampling strategy\nThe Sampes in Figure 4 and Figure A2 is randomly choose from the generated scenes. Figure 4 also show  2 samples to show variance of our method. And the results in the tables are statistic on all the generated results with the related prompts. \n\nAnd we further implement a user study to confirm the stability and diverisity of our method.\nWe increase the generated scene (randomly chosen) number from 1 to 3 for each method in the second comparison setting in order to 1) reduce individual variance and 2) check diversity of different generation system.\nWe ask the participants to choose:\n\n- Which method do you prefer?\n- Which method has greater diversity?\n\n| Method   |   w/ LayoutGPT | w/ I-Design | w/ Holodeck |\n|-----------|--------|---------------|-----------------|\n| Preference    | 94.30% | 91.40%        | 87.40%          |\n| Diversity | 95.60% | 98.90%        | 90.00%          |\n\nResults shows our methods has greater diversity while also stand for higher preference, showing the stability of our method."}]

Author rebuttal:

Meta-review: The paper presents an LLM-driven method for creating synthetic 3D scenes. The reviewers appreciate the proposed pipeline which closes the loop in ensuring physical plausibility and includes self-evaluation (during synthesis). While some concerns were raised during the discussion period, these were mostly addressed. The main outstanding issues is the use of the LLM as a final evaluation metric for reporting quantitative performance. It was noted that while there is recent precedence in the literature for LLM-based evaluation, the approach is fundamentally flawed especially where the same evaluation metrics are used during refinement of the generation process. The Area Chair agrees with this assessment. The end result being that the metrics merely test the correctness of the iterative process rather than the overall (independent) performance of the model itself. The authors should clearly identify this shortcoming in the paper. There are also minor concerns around naming of performance metrics that should be clarified in subsequent revisions of the paper. Overall, however, the contributions of the work were appreciated and warrant publication.
Decision: Accept (poster)