PhyScensis: Physics-Augmented LLM Agents for Complex Physical Scene Arrangement
Authors: Yian Wang
Han Yang
Minghao Guo
Xiaowen Qiu
Tsun-Hsuan Wang
Wojciech Matusik
Joshua B. Tenenbaum
Chuang Gan
Venue: ICLR 2026 Poster (2026)
Keywords: Physical Scene Generation
Primary area: applications to robotics, autonomy, planning

Abstract: Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement, it often overlooks the physical relationships between objects (e.g., contact, support, balance, and containment), which are essential for creating complex and realistic manipulation scenarios such as tabletop arrangements, shelf organization, or box packing. Compared to classical 3D layout generation, producing complex physical scenes introduces additional challenges: (a) higher object density and complexity (e.g., a small shelf may hold dozens of books), (b) richer supporting relationships and compact spatial layouts, and (c) the need to accurately model both spatial placement and physical properties.
To address these challenges, we propose PhyScensis, an LLM agent-based framework powered by a physics engine, to produce physically plausible scene configurations with high complexity.
Specifically, our framework consists of three main components: an LLM agent iteratively proposes assets with spatial and physical predicates; a solver, equipped with a physics engine, realizes these predicates into a 3D scene; and feedback from the solver informs the agent to refine and enrich the configuration. 
Moreover, our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters (e.g., relative positions, scene stability), enabled through probabilistic programming for stability and a complementary heuristic that jointly regulates stability and spatial relations.
Experimental results show that our method outperforms prior approaches in scene complexity, visual quality, and physical accuracy, offering a unified pipeline for generating complex physical scene layouts for robotic manipulation.

Published as a conference paper at ICLR 2026
PHYSCENSIS: PHYSICS-AUGMENTED LLM AGENTS FOR COMPLEX PHYSICAL SCENE ARRANGEMENT
Yian Wang Han Yang!*
Minghao Guo
Xiaowen Qiu
Tsun-Hsuan Wang" Wojciech Matusik
Joshua B. Tenenbaum Chuang Gan 1.4f
UMass Amherst Genesis AI MIT
*MIT-IBM Watson AI Lab
(yianwang, hanyang, chuanggan @umass.edu
guomh20148gmati
(xiaowen. qiu, (et.n/soooneeneess-eaa company
[wojeiech, jbti edu
ABSTRACT
Automatically generating interactive 3D environments is crucial for scaling up robotic data collection in simulation. While prior work has primarily focused on 3D asset placement, it often overlooks the physical relationships between objects (e.g., contact, support, balance, and containment), which are essential for creat- ing complex and realistic manipulation scenarios such as tabletop arrangements, shelf organization, producing complex physical scenes introduces additional challenges: (a) higher object density and complexity (e.g.. small shelf may hold dozens of books), (b) richer supporting relationships and compact spatial layouts, and (c) the need accurately model both spatial placement and physical properties. To address these challenges, we propose PhyScensis, an LLM agent-based framework pow- ered by a physies engine, produce physically plausible scene configurations with high complexity. Specifically, our framework consists of three main compo- nents: an LLM agent iteratively proposes assets with spatial and physical pred- icates; solver, equipped with a physics engine, realizes these predicates into 3D scene; and feedback from the solver informs the agent to refine and en- rich the configuration. Moreover, our framework preserves strong controllability over fine-grained textual descriptions and numerical parameters (e.g., relative po- sitions, scene stability). enabled through probabilistic programming for stability and a complementary heuristic that jointly regulates stability and spatial relations. Experimental results show that method outperforms prior approaches in scene complexity, visual quality, and physical accuracy, offering a unified pipeline for generating complex physical scene layouts for robotic manipulation. More quali- tative results are on physcensis, perssennissgithubio...
box packing. Compared to classical 3D layout generation,
INTRODUCTION
The creation large-scale training data has become a key driver in advancing robotics and embodied AI. A prominent line of recent work highlights simulation-based data generation effective strat- egy (Deitke et al., 2022; Wang et al., 2023b;a; Ha al., 2023; Dalal et al., 2023; Wang et al., 2024b; Pfaff et al., 2025; Wang et al., 2025), as simulation offers a scalable and cost-efficient means of data collection. Among the various forms of data required for training embodied agents, constructing diverse and complex environments for robot manipulation tasks is particularly crucial (Wang al., 2023b; Yang et al., 2024c; Wang et al., 2024b; Pfaff et al., 2025).
Prior efforts have approached this problem using procedural, rule-hased generation (Deitke al., 2022; Raistrick et al., 2024), where human experts manually design rules for scene construc- tion. While effective in controlled cases, these methods are inherently constrained to the sce- narios envisioned by the designers. Another line of work trains models on large-scale 3D scene
"quual contribution Corresponding author

---

Published as a conference paper at ICLR 2026
(a) Complexity
(3)
(4)
(b) Controllability
stacks of books, each with 3-4
An organized dining table up for people.
compact
books where the
books in each
stack are visibly
different from
each other but all
have roughly the same color within
messier
each stack.
(c) Diversity
deck afflodiigg scene hand truck and stack boxes.
sink with stacks plates and other tableware.
working desk computer science PhD student.
hox full of books.
Figure I: We present PhyScensis, an agentic framework that incorporates physics engine for phys- ical scene arrangement. PhyScensis is: (a) capable of generating complex scenes with high object density and intricate physical interactions; (b) highly controllable, with with strong text-following abilities; and (c) adaptable diverse, open- vocabblayyy scenarios.
datasets (Paschalidou et al., 2021; Feng et al., 2024: Tang et al., 2023; Yang et al., 2024b; Pfaff al., 2025). Although such methods enable learning-basee based generalization, they remain limited by the availability of datasets (e.g., Fu et al. (2020)), which provide only sparse coverage of detailed small-object placement. Moreover, both rule-based and data-driven approaches typically operate on fixed asset libraries, hindering their ability to support open-vocabulary generation.
Recent work has also explored LLM-based agent frameworks for open-vocabulary scene genera- tion (Wen et al., 2023; Yang et al., 2024c; Wang et al., 2024b; Ling et al., 2025; Wang et al., 2023b; Sun al., 2025b; Pun al., 2025: Gumin et al., 2025; Gu al., 2025; Dong et al., 2025; Abdel- reheem et al., 2025). Some of these methods leverage priors from generated images to construct al., 2025); however, they often lack fine-grained control and suffer from occlusion issues inherent to 2D image generation. Other approaches directly prompt LLMs or VLMs to predict object placements (Wang et al., 2023b; Sun al., 2025b; Abdelreheem al., 2025), but this requires strong 3D spatial and geometric rea- soning capabilities, which remain a challenge for current models. third strategy uses LLMs to generate spatial predicates that are then resolved by external solvers (Wen al., 2023; Yang al., 2024c; Gumin et al., 2025; Pun et al., 2025). Nevertheless, these methods typically (1) rely on sim- plified collision avoidance mechanisms (e.g., axis-aligned bounding boxes) that operate primarily
3D scenes (Wang et al., 2024b; Ling al., 2025; Dong al., 2025;
2

---

Published as a conference paper at ICLR 2026
in 2D, and (2) lack feedback self-correction loops, thereby limiting their scalability in crowded scenes and their capacity to capture the natural complexity of real-world placements. Finally, ex- isting approaches fail to account for the rich physical interactions found in real-world 3D scenes, such as stacking, containment, and support relationships, along with the detailed physical properties of objects. These aspects essential for generating physically plausible layouts. Although some works (Wang et al., 2024b; Ling et al., 2025; Sun et al., 2025b) can produce visually convincing stacking, their results do not guarantee physical accuracy, often leading object penetrations or unstable configurations.
To address these challenges, we propose an agent-based framework, coupled with a physics engine, that generates physically accurate scenes while also taking objects' physical properties into consid- eration. Specifically, our system consists of three main components: (1) an L.M agent that takes scene description as input and proposes set of objects along with spatial and physical predicates, (2) a solver equipped with physics engine that realizes these predicates into concrete scene param- eters, and (3) feedback system that analyzes the generated scene and provides corrective signals to the LLM agent for refinement or further generation.
Our framework offers several benefits. First, the core of the framework is physics simulator with heuristic methods that generates scenes with a high degree of naturalness and physical plausibility, well complex stacking behaviors (Fig. la). By allowing objects to interact and settle under sim- ulated physical forces, our method can produce intricate and realistic placements. also leverages probabilistic programming techniques (Wang et al., 2024a) to measure the stability of the place- ment, allowing optimization towards user-intended corner cases, such as unstable stacks (Fig. la.1) or partially supported placements (Fig. la.2). Second, the LLM-powered agentic framework allows for strong controllability and text-following ability, capable of following highly detailed instructions (Fig. lb.1) adjusting properties such as the compactness and messy level of scene (Fig. 1b.2) Third, our feedback system enhances the agent's ability to perceive and improve scenes through multiple modalities: grammar checks, failure reason detection, empty-space identification, VQA- based metrics (Lin et al., 2024) for evaluating clutter and organization, and stability assessments from the physics engine and probabilistic programming. Together, these components enable effi- cient self-correction and fine-grained control over scene arrangement.
Besides the ability to generate physically accurate and structurally complex scenarios for robot ma- nipulation, approach can generate diverse scenes free from training data requirements. As illus- trated in Fig. Ic and Fig. 5. it can produce diverse environments across settings such as boxes, kitchen setups, shelves, tabletops, and floor arrangements. Experimental results show that our method outperforms prior approaches in visual quality, semantic correctness, and physical accuracy. Furthermore, we demonstrate its utility for robotic learning by automatically collecting demonstra- tion data in generated scenarios and training policy that successfully transfers to unseen, human- designed setups- highlighig the potential of our framework for automatic data generation in em- bodied AI.
We summarize our main contributions as follows:
We propose PhyScensis, an agentic framework that leverages procedural predicates to generate interactive physical scenes, together with comprehensive feedhack system that enables the agent to perceive its environment more effectively and iteratively refine scene arrangement.
We incorporate physics simulation into the arrangement process, ensuring natural object place- ment, rich stacking behaviors, and high physical plausibility.
We demonstrate that our framework significantly outperforms prior methods in generating com- plex, physically plauuibe scenes. Through quantitative and qualitative experiments, we validate that our approach produces arrangements with level intricacy and naturalness that is unattain- able by models lacking physics-based generative process.
2 RELATED WORKS
2.1
INTERACTIVE PHYSICAL SCENE GENERATION
Numerous recent works have studied automatic scene generation for embodied AI, spanning navi- gation and manipulation applications. One line of research focuses procedural generation with
3

---

Published as a conference paper at ICLR 2026
manually defined rules (Deitke et al., 2022; Raistrick et al., 2024), which provides precise control but relies on expert-designed heuristics. Another direction trains generative models, such as trans- formers diffusion models, on large-scale 3D scene datasets (Paschalidou et al., 2021; Feng al., 2024; Tang et al., 2023; Yang et al., 2024b; Pfaff et al., 2025), capturing common spatial patterns from data. LLLM-based approaches have also emerged, including those that generate symbolic con- straints for external solvers (Wen et al., 2023; Yang et al., 2024c; Pun et al., 2025; Gumin et al., 2025), or directly predict placements through prompting (Wang et al., 2023b; Abdelreheem al., 2025; Sun et al., 2025b). Meanwhile, image-driven pipelines (Lei et al., 2023; Wang et al., 2024b; Ling al., 2025; Gu et al., 2025; Dong et al., 2025) exploit 2D generative priors such as image gen- eration (Podell et al., 2023; OpenAI, 2025), depth estimation (Yang et al., 2024a; Ke et al., 2024), object recognition (Liu et al., 2024) and segmentation (Kirillov et al., 2023) to construct layouts. Together, these approaches have established diverse strategies for scene synthesis, from rule-based systems to data-driven and LLM-guided frameworks. Our work builds on this landscape by empha- sizing richer physical interactions and properties, which remain less explored in prior efforts.
2.2 PHYSICALLY ACCURATE GENERATION
There is a growing body of work on incorporating physics into 3D modeling, Some approaches leverage video input, where temporal context facilitates the inference of physical properties such as material parameters (Zhong et al., 2024) and geometry (Li et al., 2022). Others follow a two-stage pipeline reconstructing object geometry from multi-view images, then applying physical simulations to the recovered shape (Feng et al., 2023; Xie et al., 2023; Mezghanni et al., 2021b;a). Another direction attempts to infer physical properties directly from static images using data-driven models to estimate attributes like shading, mass, and material (Zhai et al., 2024; Bell et al., 2014; Standley et al., 2017). Guo et al. (2024) further introduces the notion of physical compatibility for single- object modeling from single image. In contrast, our work targets physically accurate generation for the 3D scene, which involves multi-object interaction and global contact constraints, making the problem inherently more challenging.
METHOD
Starting from empty state with supporting surface, our method generates complex physical scene layouts from a natural language prompt. The generated scene configuration includes both the 3D placement of assets and their associated physical properties. As illustrated in Fig. 2, our pipeline consists of three stages: (a) an LLM agent takes a user prompt as input and generates a set of spatial and physical predicates, together with object descriptions for retrieval; (b) solver computes the final scene configuration using sample-based constraint solver for spatial predicates and physics engine for physical predicates; (c) feedback system reports whether the scene was successfully solved or indicates possible causes of failure, which enables the LLM agent to iteratively refine and regenerate predicates.
3.1 ASSET DATASET
We build our 3D asset dataset with BlenderKit (BlenderKit Contributors, 2025), using Chat- GPT (Hurst et al., 2024) to annotate the front direction, text description, supporting probability, and ranges of physical properties such as mass, friction, and center-of-mass shift. During scene generation, if there is no match for a given object description, we employ text-to-3D pipeline to generate new assets. More details Appendix A.4.1.
3.2 PREDICATE DEFINITION
We define a set of spatial and physical predicates that govern the placement and orientation of objects in a scene. Spatial predicates specify 2D positional or rotational relations the z-y plane, while physical predicates capture more complex 3D interactions such as stacking, supporting, or containment. These predicates are resolved by the solver to update object positions and orientations consistently. Each predicate may also include parameters (e.g., distance, alignment offsets), which are initialized by the agent and optimized by the solver. The complete definitions and formatting conventions for each predicate provided in Appendix A.4.2.
4

---

Published as a conference paper at ICLR 2026
User Prompt
Feedback System
Solver
"An organized study table for middle school student."
Guerna heelbck
inctuloskk 1 slasing rotational conaocsnn.
Spatial Solver
lask_j Feller off the (back) side of Lss Lable. tressnenent Seraing:
the right of Leptop_e
softresdbac
Collision
Procedural Predicates
Physics Solver
Spatial Predicates
LLM-Agent
Stacking
Pecttional
latt-of, right
Rotationa:
fassng-to, Facing-front,
symmerry
syenetry-storg
Sampled Initiative
Greasing
group, capt-greun
Clartspp slin silver laptop with sini wiisa desage" Plantsp tAZLE"
Physics Engine
y": 2.61].
Physical Predicates
"rcet",
Spec linee acement
aeditersized retebeok with Lired pages ['rotetoo "HONT-0F"
[rooteton F",
Stacdeg::
l'Etstarce" 6x181.
('roteton e', MacINs-SanE-dS "laptoo e", E3)
Free Rlasement: p:ece-arje1e77
:
Random & Compact Placement
Figure 2: Our framework consists of three components: (a) an LLM agent that takes a user prompt and generates spatial and physical predicates, along with object descriptions for retrieval; (b) solver that computes the final scene using a physics engine for physical predicates and a sample-base constraint solver for spatial predicates; and (c) feedback system that reports success diagnoses failure, allowing the LLM agent to iteratively refine and regenerate predicates.
Spatial Predicates. Spatial predicates define object placement in the z-y plane and include relative positioning, alignment, symmetry, orientation, and grouping:
2D Positional: left seftt/rigg//fro/////eeee
places one object relative to another along the I or y axis with specified distance. place-on-base places an object on the supporting surface with specified or randomized and coordinates.
Alignment: al
http://aa...........neeeeeeeeeee aligg-ceeteer1r// constrain
al
bounding box edges centers.
Rotation:
facing-to, facing sameinn-sam-aaae and random- rot determine
faci oepeinggoopiiiooooo te-to, byyent-by-reeattiie object's yaw orientation relative to another object or the global coordinate system.
Symmetry: symmerry-alee places an object symmetrically with respect to reference object and axis.
Grouping: group creates a virtual group objects with a defined anchor, while instantiates new group by duplicating an existing and preserving its relative structure.
Physical Predicates.
Physical predicates, as shown in the examples Fig. 3, capture 3D interac- tions, including supporting, containment, and stacking:
Container Placement: place-in in handles placing batch of objects into container.
Stacking:
pl place-on places a new object on an existing object with physical plausibility, allow- ing the user to specify the relative position, support ratio (the ratio of the contact area to the object's bottom area), and stability.
Free Placement: pl sacce-anywhers assigns a random, supported, and penetration-free position when explicit constraints are unnecessary.
These predicates enable both fine-grained spatial reasoning and the generation of physically plausi- ble arrangements. Spatial constraints ensure consistent 2D positioning and orientation, while phys- ical predicates capture richer 3D relationships such support, stacking, and containment.
5

---

Published as a conference paper at ICLR 2026
PLACE-IN
STACKING-Position Control
STACKING-Stablity Control
x_offset 0.05 y_offset
support 0.4
stabilåty 0.00/
stability = 0.85
1.00
Figure 3: Examples of placements generated by physical solvers.
Scoring with Constraints
Scones from low hig
Supporting Surfeee
book &
sipsort,
subiikee
Current
Sampling
L Voxelization
Q Grid-Based Search
Sampling with Physics Engine
Crl Nonoupprrtiv
Capparting
Diject Gndd
Contact Surface
Center Maos
Eamnox hal
Plepio Eegine
Unsupported
stabilly
L Meleooo
suabily
Object Grid
Figure 4: The stacking generation pipeline uses an accupancy-grid-Bae heuristic to efficiently compute candidate placement locations via grid search, which then ranked by requirements. A physics simulator verifies physical validity (e.g., whether an object will fall), and probabilistic programming further assesses stability, enabling control over the robustness of valid states.
3.3
SOLVER
Our solver consists of two components: spatial solver and physical solver. The spatial solver resolves spatial predicates to determine 2D positions and orientations of objects, while the physical solver addresses physical predicates to construct complex supporting and stacking behaviors, with first apply the spatial solver to determine the 2D
the help of physics engine. In practice, placements for objects specified to be placed on the supporting surface (such as a table), and then apply the physical solver solve for objects with 3D placements.
Spatial Solver. The spatial solver focuses on objects defined by spatial predicates that can be resolved deterministically in the z-y plane, which includes all objects placed the supporting surface. The spatial solver first determines whether an object is fully solved, which means its x- coordinate, y-coordinate, and yaw orientation are determined by the predicates' parameters can be inferred from the predicates. If the object is fully solved, an initial position and orientation can be assigned it as a candidate placement. Otherwise, the feedback system alerts the LLM agent, prompting it to provide additional predicates.
To evaluate candidate placements, we compute the 2D convex hull of each asset and use both (i) convex-hull overlap area between objects and (ii) the distance between an object's center and the table boundary as penalty terms. Compared to axis-aligned bounding boxes, convex-hull overlap provides more precise collision checks while remaining significantly faster than full 3D mesh inter- section tests, thereby enabling a richer set feasible configurations.
Inspired by prior optimization methods (Gumin et al., 2025), we iteratively refine the parameter set of all spatial predicates by optimizing one parameter at a time. If the resulting penalty falls below a predefined threshold, the placement accepted penetration-fre free and within the table boundaries. Otherwise, the solver fails to converge within a fixed number of steps, the case reported to the feedback system unsolvable under the given predicate set.

---

Published as a conference paper at ICLR 2026
Physical Solver. Once spatial placements established. we resolve physical predicates using the physical solver. Different types of physical predicates are handled as follows:
For place-in, adopt a physics-based packing strategy similar Blender's physics placer Bal (2024); objects are initialized above the target container and released with forces to settle into penetration-free, physically plausible poses.
For place-on and Salace-anywheee we adopt an vecupancy-gride -rid-bbaeddddd heuristic combined with a physics engine (Fig. 4). Both the current scene and the candidate asset are voxelized into occupancy grids, and feasible placements are identified as grid positions that are penetration-free and whose projected center of mass lies within the supporting convex hull. We then score the filtered candidates based on predicate parameters. For example, in Fig. 4, the predicate specifies placing cup on a book with a bottom-area support ratio of 0.4 and aims for an unstable placement, so candidates near the book's edges receive higher scores. While this heuristic provides efficient priors, it is limited by grid resolution and the lack of continuous physics. To ensure physical plausibility, we subsequently validate sampled candidate placements using a physics engine, where only objects with no large displacement after the simulation are regarded as successfully placed. More details are available in Appendix A.4.3.
Moreover, we employ probabilistic programming techniques (Wang et al., 2024a) to measure the stability of placement by sampling perturbations around it. A placement is considered more stable nearby perturbations also result in valid, balanced states. Specifically, we sample the 3D position, Euler angles, mass, center-of-mant shift, and friction coefficient around the current state using normal distribution within the range of that object. This allows us to compute the probability of each state being sampled. We then run simulations in the physies engine determine whether the system remains stable or falls, and finally estimate the overall stability probability of the state using a Bayesian approach. More details are explained in Appendix A.4.4. With this framework, we can choose to further repeatedly optimize those parameters to a more unstable state by choosing the unstable yet still not falling configurations each iteration, achieving the extremely unstable placements as in Fig. 3.
3.4 FEEDBACK SYSTEM
After processing with the solver, we provide feedback to the LLM agent to close the loop and enable iterative refinement of the scene. The feedback serves three primary purposes: detecting errors, diagnosing unsolvable cases, and evaluating successfully generated scenes.
Grammar Feedback. First, we check whether the generated predicate set is grammatically valid and fully parsable. This includes verifying that each object is fully solved according to the criteria defined earlier. If any predicate is ill-formed explicit feedback identifying which objects and predicates are problematic.
object remains unsolved, the system returns
Solver Failure Feedback. If the grammar is correct but the solver fails to find a valid configura- tion, we provide diagnostic feedback describing which objects invalid and why. Possible failure modes include object penetrations, falling outside the supporting surface, or inability to find a feasi- ble stacking pose. addition, we estimate the crowding level of the scene and heuristically identify empty areas. This information is then communicated to the agent in natural language, e.g.. "There an empty region behind the laptop on the left side of the table.". Such feedback helps the agent adjust its predicates and resample placements in less congested regions.
Success Feedback. When the solver successfully produces valid scene, we return evaluation metrics to guide further refinement. These include: (I) Stability: a stability score estimated by the physics engine together with the probabilistic programming techniques; (2) Visual Quality: VQA score assessing whether the scene appears organized, cluttered, or messy; and (3) Heuristic Measures: additional statistics such as surface coverage, compactness. and the number of objects placed. The agent incorporates these measurements to decide whether the current scene is satisfac- tory or whether further adjustments and resampling necessary.
7

---

Published as a conference paper at ICLR 2026
4 EXPERIMENTS
To validate the effectiivnness of PhyScensis we compare it with state-of-the-arr baselines, both qual- itatively and quantitatively. We also conduct a robotic experiment validate the effectiveness of our generated scenes for robot manipulation policy training. ablation study, we evaluate the ef- fectiveness of the feedback system within our agentic framework, as well our placement design. We further present an analysis on stability control in Appendix A.5.4.
4.1 BASELINES
We compare our method with open-vocabulary scene generation approaches capable of placing ob- jects in local scenarios. Specificallyy, we consider 3D-Generalist (Sun et al., 2025b), which uses Molmo (Deitke et al., 2025) to iteratively point to 2D pixels for object placement, and Archi- tect (Wang et al., 2024b), which leverages image inpainting to generate placements, followed by recognition, segmentation, and depth estimation to infer 3D positions. Both baselines require asset retrieval; to ensure fair comparison, mentation details are provided in Appendix A.5.1. Additionally, in Appendix A.3 we compare our work with LayoutVLM (Sun et al., 2025a), which is the state-of-the-as room level scene genera- tion method, and ClutterGen (Jia & Chen, 2024), which is dedicated to generating cluttered object arrangements supporting surface.
use the same asset dataset our method. Further imple-
Metrics
Method
VQA Score f
GPT Ranking 4
Settle Distance |
Reaching t
Placing
Architect
0.493 + 0.392
2.607 0.673
0.405 + 0.471
3/10
0/10
3D- Generalist
0.578 + 0.399
1.946 0.731
0.033 = 0.048
4/10
1/10
Ours
0.704 0.425
1.429 0.562
0.003 0.008
9/10
3/10
Table 1: Quantitative comparison of PhyScensis with baselines.
Metrics. We evaluate generated scenes using the following metrics: VQA Score (Lin et al., 2024): VQA model estimates the probability that the rendered image matches the input caption; GPT Ranking: GPT ranks rendered scenes for given caption, and we report the average rank as score; Settle Distance: after simulating for a fixed number of timesteps, we compute the average displacement of objects, indicating physical stability.
Analysis. As shown Table 1, our method outperforms both baselines across all metrics. For 3D-Generalist (Sun et al., 2025b), scene quality is limited by the VLM's relatively weak spatial reasoning ability. The VLM is capable of simple pointing tasks, such as identifying an object's po- sition, but struggles to effectively reason about suitable placement for given object. For Archi- tect (Wang et al., 2024b), performance depends heavily on the quality of inpainted images produced by Stable Diffusion XL (Rombach al., 2021). However, the generated images are often imperfect, resulting in suboptimal placements and lower quality scores.
Neither baseline adequately addresses physical accuracy. 3D-Generalist applies basic collision avoidance but does not enforce stability. As a result, it struggles with complex stacking scenarios (e.g., placing chopsticks on a bowl, shown in Fig. 1). Architect relies solely on depth estima- tion, frequently causing inter-object penetrations, which might cause an explosion of the simulation. These shortcomings are reflected in the higher settle distance reported in Table 1. Furthermore, both baselines depend strongly on 2D visualizations, making them unreliable for cluttered scenes with significant occlusions. Architect performs only a single inpainting step for local scenarios, while 3D-Generalist requires repeated queries for each placement, limiting its scalability. As illustrated in Fig. 5, our method consistently produces more complex and cluttered scenes, with the ability to stack objects and iteratively expand scene complexity.
4.2 ABLATION STUDY
We conduct an ablation study to evaluate the contribution of our feedback system in correcting failure cases. We measure the average number of resampling attempts required for self-correction
8

---

Published as a conference paper at ICLR 2026
Architect
3D-Generalist
Ours
dining table setting up for four.
kitchen set placement.
A hobbyist's desk with train parts, tools and paints.
A basket with two stacks of books.
A shelf many tools like drills, wrenches, toolboxes and motors.
A dock unloading scenario, with pallets, crates, cargo boxes and safety cones.
Figure 5: Qualitative comparison of PhyScensis with baselines for different generating scenarios.
Metries
Metrics
Method
Retry Times
Time-cost
Ours w/o feedback
1.69 = 1.92
132. 29 = 78.38
Method
VQA Score
GPT Ranking
Settle Distance
report
59.19
Random
0.415 + 0.363
2,706 +: 0.666
0.004 0.003
Ours Visual
0.95 = 0.91
120.65 = 53.62
LLM-Only
0.502
1.882 0.676
0.154 0.133
55.53
Ours
0.704 0.425
1.411 0.492
0.003 0.008
Table 2: Comparison with ablated versions time-cost and retry times.
Table 3: Comparison with ablated versions seene quality.
and compare the following variants: Ours w/o feedback: Removes the feedback system entirely, providing only binary success/failure signal; Ours w/o report: Retains most feedback but excludes the reporting of empty regions; Ours Visual: Augments the full framework with visual feedback

---

Published as a conference paper at ICLR 2026
for failure correction. The quantitative results are presented in Table 2. A detailed analysis of these results is deferred to Appendix A.5.2.
To evaluate the contribution and effectiveness of our designed predicates and solvers, conduct another ablation to compare our full method against two baselines: Random: After an LLM pro- poses a list of objects, we place them at random, collision-free locations and orientations the table using our spatial solver. LLM-Only: An LLM directly proposes objects along with their specific locations and orientations, similar to LayoutGPT (Feng et al., 2024). We use the same metries as in Sec. 4.1: : VQA Scores, GPT Ranking, and Settle Distance. The results are summarized in Table 3, with full analysis available in Appendix A.5.3.
4.3 ROBOT EXPERIMENT
To evaluate the usefulness of generated scenes for robotics, we conduct an imitation learning exper- iment. We constrain the scene distribution with the prompt "a dining table set up for people" and benchmark the task: "pick up the leftmost cup and place it on the rightmost plate." " For simplic- ity, we fix the cup and plate assets across scenes. We collect 300 scenarios for each method (ours and the baselines) and record one demonstration trajectory per scenario, resulting in training datasets for diffusion policy (Chi et al., 2024). For evaluation, we additionally collect 10 scenarios designed by humans and measure two metrics: reaching success rate (the robot arm reaches the correct cup) and placing success rate (the full trajectory succeeds).
As shown in Fig. 11 in Appendix, policies trained on our generated data generalize effectively to unseen human-designed scenes. Quantitative results in Tab. I further demonstrate that method achieves higher success rates than the baselines, together with Fig. 12, indicating that our generated scenes better approximate real-world distributions.
4.4
USER STUDY
We conducted a study comparing method against two main baselines: Architect (Wang et al., 2024b) and 3DD-Generalist (Sun et al., 2025b). Specifically, we selected 6 prompts and ran- domly sampled one result for each of the three methods, creating total of 18 evaluation cases. For each case, we asked users to rate the generated scene on a scale of 1-5 across three dimensions: text alignment, naturalness & physical plausibility, and complexity. We collected responses from 20 participants; the results are shown in Table 4.
Baseline
Match with Text
Naturalness Physies f Complexity
Architect Generearal
2.68
2.65
2.69
2.54
2.72
3.04
Ours
4.04
3.98
3.82
Table User Study Results. We report the mean scores from user evaluations (scale 1-5).
Our method significantly outperforms the other two baselines, which is consistent with our quanti- tative metrics. This result also aligns with our qualitative observations: Architect estimates object layout directly from inpainted images, often leading object penetrations and implausible physics. Meanwhile, 3D-Generalist employs a VLM to output placement positions in pixel space. However, the VLM's limited spatial reasoning capability fails yield reasonable layouts for complex prompts, resulting in this baseline receiving the lowest scores for text alignment.
CONCLUSION
In this work, we tackle the task of generating complex 3D scenes with rich physical interactions, an area that has not been thoroughly explored in prior work. To address this challenge, we propose PhySScensis, an LLM-based agentic framework augmented by physics engine. The LLM agent proposes a list of assets and placement predicates based on user's description, ensuring strong controllability, while the solver powered by physics engine ensures natural and physically accurate placements. A feedback system further enables scene refinement and iterative generation. Experi- mental results show that our framework can generate diverse physical scenes with complex stacking behaviors and natural placements, outperforming prior work both qualitatively and quantitatively.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
We thank the anonymous reviewers for their helpful suggestions. This work was supported in part by MURI grant 500014241217 and NSF grants 245000 and IIS-2404386. This work was partially completed while Yian was interning at Genesis Al.
REFERENCES
Ahmed Abdelreheem, Filippo Aleotti, Jamie Watson, Zawar Qureshi, Abdelrahman Eldesokey, Peter Wonka, Gabriel Brostow, Sara Vicente, and Guillermo Garcia-Hernando. Placeit3d: Language-guided object placement in real 3d scenes. arXiv preprint arXiv:2505.05288, 2025.
2,
Genesis Authors. Genesis: A universal and generative physics engine for roboties and beyond. URL
https://github.
2024. 26
Pamir tool.
Bal.
Physics
placer:
Blender
addon
dynamic
placement
https: / /tmpippaiiiia rbal. www.:et..ttttttioooooooooooo.t wtt.stttttiocccoomoooooomom wtt.:/ccccceeeeeeeeeeeeeeee store/NrPYooooo physi physics-ples casiccc-ppeeeee -nn pawpmmmm-oplaccmmeetttoooo 2024. Blender
add-on; proprietary. 7
Sean Bell, Kavita Bala, and Noah Snavely. Intrinsic images in the wild. ACM Transactions Graphics (TOG), 13- 2014.
BlenderKit Contributors. Blenderkit: Free 3d assets library for blender, 2025. URL https: // //www. bl enderkki com. Accessed: 2025-09-17. 4, 20
Cheng Chi, Zhenjia Xu, Siyuan Feng, Eric Cousineau, Yilun Du, Benjamin Burchfiel, Russ Tedrake, and Shuran Song. Diffusion policy: Visuomotor policy learning via action diffusion. The Inter- national Journal of Robotics Research, 2024. 10
Murtaza Dalal, Ajay Mandlekar, Caclan Garrett, Ankur Handa, Ruslan Salakhutdinov, and Di- eter Fox. Imitating task and motion planning with visuomotor transformers. arXiv preprint arXiv:2305. 16309, 2023. 1
Matt Deitke, Eli VanderBilt, Alvaro Herrasti, Luca Weihs, Kiana Ehsani, Jordi Salvador, Winson Han, Eric Kolve, Aniruddha Kembhavi, and Roozbeh Mottaghi. Procthor: Large-scale embodied ai using procedural generation. Advances in Neural Information Processing Systems, 35:5982-
5994, 2022. 1,4
Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Moham- madreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models. In Proceedings of the Com- puler Vision and Pattern Recognition Conference, pp. 91-104, 2025. 8, 27
Wenqi Dong, Bangbang Yang, Zesong Yang, Yuan Li, Tao Hu, Hujun Bao, Yuewen Ma, and Zhaopeng Cui. Hiscene: creating hierarchical 3d scenes with isometric view generation. arXiv preprint Sttt::2504.33777. 2025. 2, 4
Weixi Feng, Wanrong Zhu, Tsu-jui Fu, Varun Jampani, Arjun Akula, Xuehai He, Sugato Basu, Xin Eric Wang, and William Yang Wang. Layoutgpt: Compositional visual planning and genera- tion with large language models. Advances in Neural Information Processing Systems, 36, 2024. 2, 4, 10
Yutao Feng, Yintong Shang, Xuan Li, Tianjia Shao, Chenfanfu Jiang, and Yin Yang. Pie-nerf: Physics-based interactive elastodynamics with nerf. arXiv preprint 231.:2311.13090 2023. 4
Huan Fu, Bowen Cai, Lin Gao, Lingxiao Zhang, Cao Li, Qixun Zeng, Chengyue Sun, Yiyun Fei, Yu Zheng, Ying Li. Yi Liu, Peng Liu, Lin Ma, Le Weng, Xiaohang Hu, Xin Ma, Qian Qian, Rongfei Jia, Binqiang Zhao, and Hao Zhang. 3d-front: 3d furnished rooms with layouts and semantics. arXiv preprint arXiv:201 1.00122 2020.

---

Published as a conference paper at ICLR 2026
GEOS contributors. GEOS computational geometry library. Open Source Geospatial Foundation, 2024. URL httpt:h//tiggeeeeeeeee org/. 26
Zeqi Gu, Yin Cui, Zhaoshuo Li, Fangyin Wei, Yunhao Ge, Jinwei Gu, Ming-Yo Yu Liu, Abe Davis, and Yifan Ding, Artiscene: Language-driven artistic 3d scene generation through image intermediary. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 2891-29011 2025. 2, 4
Maxim Gumin, Do Heon Han, Seung Jean Yoo, Aditya Ganeshan, R Kenny Jones, Rio Aguina- Kang, Stewart Morris, and Daniel Ritchie. Imperative vs. declarative programming paradigms for open-universe scene generation. arXiv preprint arXiv:2504.05482 2025. 2, 4,
Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Owens, Chuang Gan, Josh Tenenbaum, Kaiming He, and Wojciech Matusik. Physically compatible 3d object modeling from a single image. Advances in Neural Information Processing Systems, 37:1 17:119920-11922,, 2024.
Huy Ha, Pete Florence, and Shuran Song. Scaling up and distilling down: Language-guided robot skill acquisition. In Conference on Robot Learning, pp. 3766-3777. PMLR, 2023.
Aaron Hurst, Lama Ahmad, Daniel Kappler, Daniel Levin, Daniel Levy, David Carr, David Farhi, David Mely, David Robinson, David Sasaki, Denny Jin, Dev Valladares, Dimitris Tsipras, Doug Li, Due Phong Nguyen, Dunean Findlay, .... et al. Gpt-4o system card. arXiv preprint 24X1::2410.21276, 2024. URL http:://doi.or//// wttpo................ 21..5///a....21......... OpenAl. 4, 20
Alec Jacobson, Daniele Panozzo, et al. libigl: A simple C++ geometry processing library, 2018. https://libigl.githee github.io/. 26
Yinsen Jia and Boyuan Chen. Cluttergen: A cluttered scene generator for robot learning. In 8th Annual Conference on Robot Learning, 2024. 8, 18,
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 4
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Jiabao Lei, Jiapeng Tang, and Kui Jia. Rgbd2: Generative scene synthesis via incremental view in- painting using rgbd diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8422-8434, 2023. 4
Xuan Li, Yi-Ling Qiao, Peter Yichen Chen, Krishna Murthy Jatavallabhula, Ming Lin, Chenfanfu Jiang, and Chuang Gan. Pac-nerf: Physics augmented continuum neural radiance fields for geometry-agnoster system identification. 2022. 4
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. arXiv preprint arXiv:2404.01291, 2024. 3,8
Lu Ling, Chen-Hsuan Lin, Tsung- Yi Lin, Yifan Ding, Yu Zeng, Yichen Sheng, Yunhao Ge, Ming- Yu Liu, Aniket Bera, and Zhaoshuo Li. Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint a55iv:2505. 0283066.......... 2025. 3,
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European conference on computer vision, pp. 38-55. Springer, 2024. 4
12

---

Published as a conference paper at ICLR 2026
Mariem Mezghanni, Theo Bodrito, Malika Boulkenafed, and Maks Ovsjanikov. Physical simulation layer for accurate 3d modeling. CVPR, 2021a. 4
Mariem Mezghanni, Malika Boulkenafed, Andre Lieutier, and Maks Ovsjanikov. Physically-aware generative network for 3d shape modeling. In CVPR, 2021b. 4
OpenAl. text-embedding-3-aaae openai com/ docs /models /ttttttteebbeeeee ng- 3-rrge.. Announced alongside new embedding models; see "New embedding models and API updates". 20
Model card / API docs, 2024. URL
OpenAL. Gpt-image-1: A natively multimodal image generation model. Technical report / API documentation, 2025. URL https: httpppaafooo openai. com/ docs pescs//modlll models/ gpt image-1. Released April 2025. Accepts text and image inputs, accurately renders text, follows complex prompts, integrates world knowledge.
Despoina Paschalidou, Amlan Kar, Maria Shugrina, Karsten Kreis, Andreas Geiger, and Sanja Fi- dler. Atiss: Autoregressive transformers for indoor scene synthesis. Advances in Neural Informa- tion Processing Systems, 34:1 10:11201112222 2021. 2,
Nicholas Pfaff, Hongkai Dai, Sergey Zakharov, Shun Iwase, and Russ Tedrake. Steerable scene generation with post training and inference-time search. arXiv preprint arXiv:2505.048:1 2025. 1,2.4
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. 2023. 4
Hou In Derek Pun, Hou In Ivan Tam, Austin T Wang. Xiaoliang Huo, Angel Chang, and Manolis Savva. Hsm: Hierarchical scene motif's for multi-scale indoor scene generation. arXiv preprint arXiv:2503. 16848, 2025. 2, 4
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- PmLR, 2021. 20
Alexander Raistrick, Lingjie Mei, Karhan Kayan, David Yan, Yiming Zuo, Beining Han, Hongyu Wen, Meenal Parakh, Stamatis Alexandropoulos, Lahav Lipson, et al. Infinigen indoors: Photo- realistic indoor scenes using procedural generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21783-21794 2024. 1,4
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021. 8, 20
Trevor Standley, Ozan Sener, Dawn Chen, and Silvio Savarese. image2mass: Estimating the mass of object from its image. In Conference on Robot Learning, pp. 324-333. PMLR, 2017.
Fan- Yun Sun, Weiyu Liu, Siyi Gu, Dylan Lim, Goutam Bhat, Federico Tombari, Manling Li, Nick Haber, and Jiajun Wu. Layoutylm: Differentiable optimization of 3d layout via vision-language models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 29469-29478 June 2025a. 8, 17,
Fan-Yun Yun Sun, Shengguang Wu, Christian Jacobsen, Thomas Yim, Haoming Zou, Alex Zook, Shangru Li, Yu-Hsin Chou, Ethem Can, Xunlei Wu, al. 3d-generalist: Self-improving vision- language-action models for crafting 3d worlds. arXiv preprint 2Xiv:2507.06484 2025b. 2, 3, 4, 8. 10, 27
Jaapeng Tang, Yinyu Nie, Lev Markhasin, Angela Dai, Justus Thies, and Matthias Nießner. Dif- fuscene: Scene graph denoising diffusion probabilistic model for generative indoor scene synthe- sis. arXiv preprint arXiv:2303. 14207, 2023. 2,
13

---

Published as a conference paper at ICLR 2026
Pauli Virtanen, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Courna- peau, Evgeni Burovski, Pearu Peterson, Warren Weckesser, Jonathan Bright, Stéfan J. van der Walt, Matthew Brett, Joshua Wilson, K. Jarrod Millman, Nikolay Mayorov, Andrew R. J. Nel- son, Eric Jones, Robert Kern, Eric Larson, C J Carey, Ilhan Polat, Yu Feng, Eric W. Moore, Jake VanderPlas, Denis Laxalde, Josef Perktold, Robert Cimrman, Ian Henriksen, E. A. Quintero, Charles R. Harris, Anne M. Archibald, António H. Ribeiro, Fabian Pedregosa, Paul van Mul- bregt, and SciPy 1.0 Contributors. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261-272 2020. doi: 10. 10.1038/s41555 0222--0190066---- 26
Haoliang Wang, Khaled Jedoui, Rahul Venkatesh, Felix Jedidja Binder, Josh Tenenbaum, Judith E Fan, Daniel Yamins, and Kevin Smith. Probabilistic simulation supports generalizable intuitive physics. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 46, 2024a. 3,7
Lirui Wang, Yiyang Ling, Zhecheng Yuan, Mohit Shridhar, Chen Bao, Yuzhe Qin, Bailin Wang, Huazhe Xu, and Xiaolong Wang. Gensim: Generating robotic simulation tasks via large language models. In Arxiv, 2023a.
Yian Wang, Xiaowen Qiu, Jiageng Liu, Zhehuan Chen, Jiting Cai, Yufei Wang. Tsun-Hsuan Wang, Zhou Xian, and Chuang Gan. Architect: Generating vivid and interactive 3d scenes with hier- archical 2d inpainting. Advances in Neural Information Processing Systems, 37:67575-67603, 2024b. 1, 3, 4, 8, 20,
Yufei Wang, Zhou Xian, Feng Chen, Tsun-Hsuan Wang. Yian Wang, Katerina Fragkiadaki, Za- ckory Erickson, David Held, and Chuang Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. arXiv preprint 1,x0v22311.01455, 2023b. 1, 2,
Yufei Wang, Ziyu Wang. Mino Nakura, Pratik Bhowal, Chia-Liang Kuo, Yi-Ting Chen, Zackory Erickson, and David Held. Articubot: Learning universal articulated object manipulation policy via large scale simulation. In Robotics: Science and Systems (RSS), 2025.
Zehao Wen, Zichen Liu, Srinath Sridhar, and Rao Fu. Anyhome: Open- vocabulary generation of structured and textured 3d homes. arXiv preprint arXiv:2312.06644, 2023. 2, 4
Tianyi Xie, Zeshun Zong, Yuxin Qiu, Xuan Li, Yutao Feng. Yin Yang, and Chenfanfu Jiang. Physgaussian: Physies-integrated 3d gaussians for generative dynamics. arXiv preprint arXiv:2311.12198, 2023. 4
Lihe Yang, Bingyi Kang. Zilong Huang, Xiaogang Xu, Jiashi Feng. and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. arXiv preprint 1rXiv:2401.10891 2024a.
Yandan Yang, Baoxiong Jia, Peiyuan Zhi, and Siyuan Huang. Physcene: Physically interactable 3d scene synthesis for embodied ai. arXiv preprint arXiv:2404. 09465,, 2024b. 2
Yue Yang, Fun-Yun Sun, Luca Weihs, Eli VanderBilt, Alvaro Herrasti, Winson Han, Jiajun Wu, Nick Haber, Ranjay Krishna, Lingjee Liu, et al. Holodeck: Language guided generation of 3d embodied ai environments. In The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2024), volume 30, pp. 20-25. IEEE/CVF, 2024c. 1,
Albert J Zhai, Yuan Shen, Emily Y Chen, Gloria X Wang. Xinlei Wang, Sheng Wang, Kaiyu Guan, and Shenlong Wang. Physical property understanding from language-embedded feature fields. arXiv preprint 440i 2444:4444 2024. 4
Licheng Zhong, Hong-Xing Yu, Jiajun Wu, and Yunzhu Li. Reconstruction and simulation of elastic objects with spring-mass 3d gaussians. arXiv preprint S.XV::2403.094344 2024. 4
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A. THE USE OF LARGE LANGUAGE MODELS
In the preparation of this manuscript, we utilized LLM proofreading tool. Its application was strictly limited to checking for grammatical errors, spelling mistakes, and improving sentence clarity. The LLM did not contribute to any scientific ideas, experimental results, the core structure of the paper.
A.2 MORE QUALITATIVE RESULTS AND FAILURE ANALYSIS
We present more qualitative results on the anonymous website https://physcensis gittiu.tttt.oo.. includ- ing generated scenes, visualization videos, and examples of our synthesized manipulation data.
A.2.1 QUALITATIVE COMPARISON WITH ABLATION BASELINES
We show qualitative comparisons with our ablation baselines: Random and LLM-Only. The Ran- dom baseline places objects at random positions on the table, while the LLM-Only baseline uses an LLM to directly output object positions and orientations. Both baselines to ensure collision-free placement. From the qualitative results in Fig. 6, although the LLM--nly baseline can produce reasonable layouts, its generated scenes lack complex stacking containment behaviors, making them look less natural and diverse.
simple collision solver
Random
LLM- Onlly
Full
Figure 6: Qualitative comparison with Random and LLM-Only baselines.
A.2.2 FAILURE CASES WITH BAD RETRIEVAL.
One major factor contributing to poor generation quality is inaccurate retrieval. Although our pipeline checks the similarity between the retrieved object and the text description to filter out mis- matched assets, it fails in some corner cases. For example, in the first image of Fig. 7, the two plastic bottles belong single asset that aligns well with the description. Consequently, it is successfully retrieved; however, after optimization, this results in floating bottle. A similar issue occurs in the second example in Fig. 7. Such failure
could be mitigated by using more comprehensive filtering pipeline when constructing the asset library for retrieval.
A.2.3 FAILURE CASES OF SOLUTION NoT FOUND
During our iterative scene generation process, as the scene becomes populated with many objects, the spatial solver may fail to find collision-free solution for new set of objects. Two such cases,
15

---

Published as a conference paper at ICLR 2026
Figure 7: Examples of cases when the retrieved objects are not reasonable.
where valid placements could not be found, are shown in Fig. 8. It's common when the table is already covered with a lot of assets. Such failure cases could be mitigated by explicitly encouraging the agent to use physies predicates-for example, to stack objects or use PLACE-ANYWHERRS to automatically find valid placements and further enrich the scene.
Before
After
Figure 8: Examples of cases when spatial solver cannot find solution.
A.2.4 FAILURE CASES WITH UNSTABLE STACKING
When processing physical predicates solve for complex stacking, the scene could sometimes be unstable when too many objects are added. In this case, objects will fall wherever the new objects placed, so that no new objects can added, even though there enough space during the grid-based search. For example, in Fig. 9, the stack of plates leads to unstable simulation where future stack always falls. In such cases, our pipeline will fail stack new assets within the time limit.
A.2.5 FAILURE CASES WITH Low VQA SCORE
In Fig. 10, show some of our generated results that receive low VQA scores. Based on these results, we observe that the VQA score can, to some extent, reflect the degree of alignment between the text prompt and the generated scene. For example, the first example may have received a low score because there are no snacks in the bowl; the second, because the laptop and display do not match the profile of middle school student; and the third, because the scene is not messy enough.
16

---

Published as a conference paper at ICLR 2026
Place
Simulate
Restore & Place again
Simulate
Figure 9: Examples of unstable stack.
However, in terms of scale, the VQA score does not always reflect human perception. For instance, while the third example is indeed not messy enough, near-zero score is disproportionately low.
A living room coffee table set up game night with board game and bowls of snacks.
An organized study table for middle school student.
messy dining table after family pasta dinner.
0.32
p 0.03
p 4c-8
Figure 10: Examples of results with low VQA Scores.
Init
Reach
Grasp
Place
Test Scenes
Figure 11: An example of policy rollout.
A.3 ADDITIONAL BASELINE COMPARISON
A.3.1 LAYOUTVLM
LayoutVLM (Sun et al., 2025a) adopts a similar method for 3D scene-level layout generation, us- ing L.MSs generate proposals for object positions and relationships, followed by differentiable solver to determine the final layouts. We present additional quantitative results (Table 5) and quali- tative results (Fig. 13). Note that LayoutVLM requires a text prompt paired with a list of specified 3D assets as input. To adapt this method to our setting of text-only conditioned scene generation,
we introduce an additional pipeline that queries GPT-4o to generate a list of object descriptions, and then uses CLIP features to retrieve corresponding 3D assets from the database. To ensure fair comparison, we process our 3D asset database (also sourced from Objaverse) into the format required by LayoutVI.M.
17

---

Published as a conference paper at ICLR 2026
Ours
3D-Generalist
Architect
Visualized samples from generated scene for the robot manipulation task. The result shows that our method, while still being diverse, resembles more to the human intuitions.
Figure 12:
Metrics
Method
VQA Score f
GPT Ranking
Settle Distance
Architect
0.493 0.392
3.250 0.947
0.405 + 0.471
3D-Generalh
0.578 = 0.399
2.393 + 0.899
0.033 = 0.048
LayoutVLLM
0.648 0.446
2.643 + 1.109
0.115 + 0.238
Ours
0.704 = 0.425
1.714 0.920
0.003 = 0.008
Table 5: Additional quantitative comparison of PhyScensis with baselines, including Layout
Our method outperforms LayoutVL.M both qualitatively and quantitatively. primary limitation of LayoutVL.M is its design focus room-level layout generation. In such settings, the environment spacious relative to the furniture, allowing for large optimization search space. However, in experiments, we constrain the optimization to small area (as all objects must be placed tabletop). Under these conditions, LayoutVLM's optimization often fails when the object count is high, such in the prompt "A stack of books on the table". In these cases, the optimization process converges prematurely (e.g., at 50 steps out of 400 iterations). At this stage, physical constraints are fully resolved, and objects remain cluttered or overlapping, visible the kitchen, shelf, and basket scenarios in Fig. 13. This is also the primary reason for the high Settle Distance observed for Lay in Table 5. Secondly, although LayoutVLM claims support the stacking multiple objects, we were unable to generate more complex stacking, such as a stack of books (where assets are different) or more random stacking. In contrast, our method excels at generating these cases with the help of a physics engine. Thirdly, LayoutVLM, applies bounding box-based collision checking, similar to most prior works, while we adopt a polygon-based method and physics engine, which is more precise for crowded scenes with natural object orientations.
A.3.2 CLUTTERGEN
Previous work, CLUTTERGEN (Jia & Chen, 2024), addresses a related problem of generating com- plex object arrangements on supporting surface specified size. Their approach employs rein- forcement learning to optimize policy for placing fixed set of assets, but it does not account for semantic relationships between objects and does not support stacking or containment behavior. In contrast, method is open-vocabulary, training-free. semantics-aware, and achieves substantially improved performance.
For fair comparison, follow the evaluation protocol of CLUTTERGEN, which prioritizes clut- terness over semantic correctness. Specifically, we randomly shuffle the asset list and sequentially place objects using the PLACE-ANNYHERS predicate, cycling back the beginning of the list after
18

---

Published as a conference paper at ICLR 2026
A dining table setting up for four.
A kitchen set placement.
hobbyist's desk with train parts, tools and paints.
A basket with two stacks of bboks.
BRICHA
A shelf with many tools like drills, wrenches, toolboxes and motors.
dock unloading scenario, with pallets, crates, cargo boxes and safety cones.
Figure 13: Qualitative results for Layout using the same prompt as in Fig 5.
ClutterGen
Ours: no stack
Ours: with stack
Ours: with skip
ADTO
Figure 14:
Qualitative comparison between our method and ClutterGen (Jia & Chen, 2024) using the same asset library. We pick the best placements of ClutterGen to render.
exhausting all objects. The process terminates once object fails to be placed. We use the same set of dining grouggggruppuo assets as CLUTTERGEN and report the total number of objects successfully placed. For CLUTTERGEN implementation, trained its RL policy for 2.5 million steps before inference with it.
We evaluate method under three settings: (1) no-stack, where stacking is disabled and objects cannot be placed atop one another; (2) with-stack, where any physically stable stacking configuration is permitted; and (3) with-skip, where objects that fail to be placed are skipped, and the system
19

---

Published as a conference paper at ICLR 2026
continues attempting the remaining objects until further placements possible or a time limit reached.
A qualitative comparison is shown in Fig. 14. Even without stacking, method produces substan- tially more compact and cluttered scenes than CLUTTERGEN. Allowing stacking or skipping failed objects further increases scene density. Quantitatively, across 500 sampled scenes, CLUTTERGEN places an average of 3.57 objects, whereas our method places an average of 14.77 and 22.30 objects under the no-stack and with-stack settings, respectively, with the with-skip variant achieving even higher counts.
A.4 IMPLEMENTATION DETAILS
A.4.1 DETAILS ASSET DATASET PREPARATION
Since our method retrieves 3D assets from an existing dataset rather than generating them from scratch, pre-process the dataset to support efficient retrieval and provide the necessary annota- tions.
We build asset library using BlenderKit (BlenderKit Contributors, 2025), chosen for its high- quality and diverse assets. We construct our asset dataset by downloading 3D models from BlenderKit (BlenderKit Contributors, 2025). To define object categories, first prompt Chat- GPT (Hurst et al., 2024) to generate a list of common small household objects. For each category, we retrieve the top 30 assets from BlenderKit and annotate them with ChatGPT, resulting in a base dataset of approximately 800 assets.
For each asset, we annotate five key properties. We determine the front direction by rendering views from all four sides and asking GPT-44 (Hurst et al., 2024) to identify the correct orientation. We then generate a text description of the asset based on its name and front-view image to support semantic retrieval and estimate the asset's 's supporting probability-iee how likely it can serve as support surface (e.g., closed laptop versus an open one)-as well as ranges of physical properties such as mass, friction, and center-of sineer-ff-maee of-mass shift. Finally, compute embeddings using both OpenAl text embeddings (OpenAI, 2024) and CLIP (Radford et al., 2021) from the rendered images, enabling multimodal retrieval.
During scene generation, if retrieval fails (i.e., the text-embedding similarity falls below a thresh- old), automatically generate an image from the description using an image generation model (Rombach et al., 2021) and then call Tripo-Al's image-to-3D API obtain the corresponding mesh. The asset is scaled using epT-esttinted dimensions and then passed through the same annotation process before being added the dataset.
asset via text-to-3D pipeline. Specifically, we first generate
A.4.2 DETAILS ON PREDICATE DEFINITIONS
Here we provide the complete prompt provided to the LLM for generating placement predicates for given scene prompt. Listing presents the system prompt, which details the formatting conventions, mathematical definition, purpose, examples, and additional explanations for each predicate. It also provides notes for the agent, such as naming conventions for new objects, object solving criteria. and other grammatical rules. Listing 2 shows the complete prompting context for predicate generation, including how the scene prompt and feedback are provided to the agent.
You are a helpful agent that helps placing a scene.
Your role is to utilize the relatienships to construct whole scene.
specifically, you need to give set of objects, their textual descriptions for retrieval, and the relationship between them. Note that the root node is the supporting surface such as a table. You should be careful to keep the object you placed on the support surface (inside the range of x, y)
There might be multiple rounds of placement generating and feedbacks, the feedbacks might include error messages, or a confirmation with request to generate more.
Sections:
20

---

Published as a conference paper at ICLR 2026
1. Coordinate conventions:
Root's bbox given in explicitly, when placing objects, ensure any coordinate updates keep bbox within these ranges. Units: meters.
Front Back -x
Left = +y
Right =
2. REIATIONSHIP DEFINITIONS
when choosing relationship ["A", predicate, "B" params], apply the corresponding update to A bbox and rotation as defined; ensure A's fields are updated before using them in subsequent predicates.
2-1. 2-D SPATIAL: LEFT-OF
A.min B.max_y params ['distance"]
RIGHT-OF
A.max_y B.min_y params "distance")
FRONT-OF
A.min_x x B.nax_x params "distance")
BACK-OF
A. max B.min x params "distance")
Example: "pphoe_0", "LEFT-OF", "bottle_0", ("distance"; 0.13]
2-2. 2-D ALIGNMENT: ALIGN-CENTER-IA
A. center_y Beeterry
ALIGN-CENTER-EB
A. centerr_ B.cetter_x
ALIGN-LAFT
A. mamay_ B.max_y
ALIGN-R GHHH
A.mi mi mi H.min_y
ALIGN-FRONT FRONT
A. max x B.max X
ALIGN-BAN BACK
Aimin_x B.min_x
Example:
O""ork "ALIGN-CENTER-FSY" "plate_0", [1]
2-3. 2-D SYMMETRY:
SYMMETRY-ALONG
Purpose: Place object A symmetrically opposite to object a across ob object
C
get_info (params["""))
A.
A.ceterrxx 2 C.center_x B. center__
A.center_y 2 c.center_y B. centery_
Example:
["knife 1", USYMMETRY-ALONGE "knife_0", ("C": plate_0))
2-4. ROTATION-ONLY:
FACING-TO
dx center_ A.center_x
dy B. centery A.center_y yayaw math. atann (dy, dx) FACING-SAME SAME-AS
A. yaw = B. y.yaw
DSCING-OPPOSIEE TО
21

---

Published as a conference paper at ICLR 2026
yaw yB. math. pi)
(2*math.pi)
Example: ["monitor_o", "FACING-TO", "keyboard_0", {)]
FACING-FRONT
A-yaw
FACING-BACK
y.w math.pi pi
FACING-LEFT
yaw math.pt /
PACING-RICG
A.yaw math.pi 2
Example:
1"book_0", 0", "FACING-FRONT", "roet", [)]
RANDOM-ROT
A.yaw = random, random () * * np.pi
Examplc:
1"pen_0", "RANDOM-ROT", "roct", ())
Purpose: Place the orientation of A relative to B; Orientation depends on which side is relative to В. Like place mouse relative to keyboard utensils relative to plate.
It will autematically gives a compact placement. Good to use together with ALIGN-CENTER-IR or ALIGN-CENTER-PE rules. def overlapX (A, B)
return max (0, min (A.max_x, B.max_x) max (A.min_x, B.min_x)) def overlapł (A, return max min (A.max_y) B.max_y) max (A.min_y. B.min_y)) A.yaw = default__ya
recompute_bbox(1) (A)
aligned_scalel overlapX B) + overlapY (A, A. default_yaw math.pi
recompute__boxtfe
al alignedsscale2 overlapx (A, 8) + overlapy (A, 8)
if al aligned_scalel alig
A.yaw default_yaw
else:
A.yaw = default yaw + math.pi 2
Example:
["fork_0", 0", "ORLENT-BX--REAATVVV-SSDS "plate_0", (H)
2-5. HEIGHT DETERMINATION:
PLACE-ON-BASE
Purpose:
Position object the reot, using the root the supporting surface.
A.center_ params "x"
A.center_y params "y"| A.min_z state state("roott)) max___
Example:
["lamp_0", "PLACE-ON-BASE" "rect", { "x": 0.2, "y" 0.3 }] Other notes:
You can also set params to empty for PLACE-ON-BASE, that it will refers other predicates for and y location
PLACE-ON
Purpose: Position object A on top of object B, using as the supporting surface.
The exact position of
will be optimized to ensure physical
plausibility.
You can also specify some parameters that control the optimization.
# The position of relative to B: A.center B.center parans offset")
22

---

Published as a conference paper at ICLR 2026
A.cetter_y B.center y parans offset"] The overlap ratio of the betton of
params ["overrap"]]] area (A (A.bettem.. intersect (B.top)) area (A. bottom)
Example:
["notebook_1", "PLACE-ON", "notebook_0", ["x_offset"; 0.0, "y_offset
0.11] or
I"notebook_"", "PLACE-ON", "nctebook_ 0", ("ovrrlap: 0.8)]
Other notes:
For this relation, cannot be "root"; It must be existing object that has flat surface.
You can also set params to empty, so that A will be randomly placed on B.
PLACE-ON can be used stack multiple objects top of each other, PLACE-ON c, PLACE-ON D, etc.
A PLACE-ON When using PLACE-ON, do not use other spatial predicates 2-D SPATIAL or 2-D ALIGNMENT for A, since its position is alroady determined by B, the offsets and the overlap ratio.
FLACE-ON is processed after other objects except PLACE-ANYWEERS So predicates related the PLACE-ON object should after other predicates but before PLACE-ANYWHERE.
2-6. GROUPING:
GROUP
Purpose: Dečine a new group object that aggregates a list of existing objects.
Usage: [group_name, "GROUP", [object_id_l id_, jd ...,, !"
anchor": object_id_k!
Conventicns:
group_namd _ame must be a unique identifier starting with "group_".
The list must refer only to objects that have already been introduced.
Effect: Creates virtual/grouped entity "group_name" that represents the set of specified objects. The facing direction of the group will be defined as the facing direction of the anchor object defined by the parameter.
Example:
"GROUP", ["plate_0", "fork_0", "knife_0", " cup cup_ l"anchor"; "piate_0") [01]
dggroup_dininggset..o
This creates a group named group dining set 0 comprising the existing objects plate_00 0, fork_ 0, knife 0, and cup 0. You don't need to do other things since they' already been placed.
COFY-GROUP
Purpose: Instantiate a new group by copying the structure and relative positiens of an existing group. Usage: [new_group_name, "COPY-GROUP", gxistinng_rropp_aame, [}] Conventions:
new_group_nee name must be a fresh identifier starting with "group_".
existing wwwgggroou_mnee name must refer to group already defined.
effect: Creates a new set of objects (with new identifiers) arranged the same relative configuration useful for duplicating table arrangement (e.g., another place setting).
existing__rrou_nnae This is
Follew-up: After copying, apply spatial predicates to position the copied group in the scene. For example:
Example:
""group_diiiiggsst__"", "COPY-GROUP", "group_dining_set_.0" (H di distance": 0,4]]
"BACK-OF", "group__ininng_set_0"
["group_dining_set__] (Hl
"EACING-OPPPSITRR-T0" "group_dining_set__
"ALIGN-CENTER-IR", wwwpe_ddiinggeeeee
()
23

---

Published as a conference paper at ICLR 2026
Other notes:
The copied group will automatically have the same set of objects. For example, if sroup_dining_stte have ["plate_0", "fork_0", "knife_0", "cup_0"], then group_dining will have ("plate_0- group_dining_sst__"", group droup_dining_set_/ No need for extra predicates for individual objects in the new group, they will keep their relation the same as the copied set.
"knife_0-
2-7. SPECIAL:
PLACE-IN
Purpose:
Place object A set/group of objects A inside container B. Useful for items in baskets, boxes, per-holders, Usage: [A, "PLACE-IN", B, (}) Conventicns:
В must be container or support object that can hold items. specification 1ist of new objects by category and quantity, e.g. [["pen", 61, ["pencil", 3]1 [["pen", 3]]. In this case, the external function creates the specified items and places them.
It you are placing muitiple object into container, please make you only place-in once to put those objects into the container rather than calling separate times.
Examples:
[[["pen", 6], ["penci1" 311, "PLACE-IN", "pen_holder_o"" {)]
PLACE-ANYNHERE
Purpose:
Place object A anywhere in the scene without specifying relatienships to other objects.
The positicn is determined by external function that ensures physical
plausibility (no collision, falling, etc.).
Example:
["vase_0", "PLACE-ANWHERE"", "reot", {)]
Other notes:
Useful for free-placement items when exact position is critical. And for placenent crowded scene.
Do not use place-anywhere for group.
Because it does not depend en other objects, PLACE-ANYWHERE should appear at the end of placement round, after all relational predicates.
3. NOTES FOR THE AGENT The support surface already placed in the scene, with "root" its name.
For each object, please first give a one-sentence description for asset retrieval, in the format of list of length 2: : id", descriptions"]
object is fully solved one (min_x wwtnlleeneeeeaaeee__eee (FRONT-OF, BACK-OF, ALIGN-CENIGR-P%, ALIGN-FRONT, ALIGN-BACK, SYMMETRY-ALONG, PLACE-ON or PLACE-OR -ON--ON---EEE--A with params can confirm this), of RIGHT-OF, LLIGN-CENTER-LR, ALIGN-LEFT, ALIGN-RIGHT, SYMMETYY-ALON,,
yl (LEFT-OF, PLACZ-ON or PLACE-ON-BASE with params can confirm this),
height (PLACE-ON-BASE, PLACE-ON), and yaw (rotational direction) are determined (FACING-TO, FACING-SAME-AS, PRCING-OPPOSTTE--D) SIDE-SCALE -ALIGN, RANDOM-ROT confirm this)
Special relationships (PLACE-IN, SELF-
STACKING) can contirm everything through external functions.
24

---

Published as a conference paper at ICLR 2026
The predicate of the same object serving should be segment in list (group the predicate for that object together the sequence).
when you introduce a new object X_N, any relationship [X_,, predicate, y, ....] must refer to V that either is "root" was introduced earlier in this list. Do not refer objects that come later. Output must be a plain list of predicates, nothing else.
For example:
"apptpp_00 0", "a slim silver Laptop with minimalist design") aptop_0", 0", "PLACH-ON--ASE" "root", ("x"; 0.0, "y": 0.0)], laptop_0", "FACING-SAME-AS" "root", {)),
nnoebbook 0", "a medium-sized notebook with lined pages"] nnotebook__ _", "PLACE-ON-BASE", "root", ()), "notebock_00 "FRONT-OF", "laptop_0 0", ("distance": 0.11], "notcbook00", "RANDOM-ROT", "root", {}),
"cup0" 0", "an empty ceramic cup with handle"] "cup_0" 0", "PLACE-ON", "notebcck_ō", ("overlap": 1.0)]
["cup_0", "FACING-FRONT", "reet", ()]
objects sub-arrangements share the relative pattern (same relative distaaces//rreenaatiius)) the assistant should create a group for that pattern and use copy-group to replicate it, to ensure consistency and brevity.
two
Naming convention is (Category)_ like candle_0 candle_front.
Special predicates always comes alone to decide the placement of one object, do not couple with other predicates for one object.
Pay attention to the sequence of predicates. It should be first other predicates, then PLACE-ON predicates and the predicates related to PLACE-ON objects, and finally PLACE-ANYHHDR predicates.
Listing 1: The system prompt for generating predicates.
"role": "system",
"content"; ["(system_prompt)"
"role": "user",
"content"; g"The xy extend of the table is stable_bbox/ An The description [M(scene_prompp]V
## if previcus response and feedback exist F#
"role":
"assistant.",
"content"; ["(previoos_reeponss]
"role": "user", "content": g"There are some errors in previous response. Here's s the feedback (feedback). Flease generate a new one to fix it and try to retain the existing relationships if possible. You should still strickly follow the output format."
## end
Listing 2: The complete prompting context for generating predicates.
25

---

Published as a conference paper at ICLR 2026
A.4.3 DETAILS ON PLACE- ON AND PLACE- IMPLEMENTATION
We implement the solver for place on and place place- anywhere with an occupancy-grid--anne heuristic augmented by a physics engine. We convert objects and the current scene into occupancy grids using libigl (Jacobson al., 2018), with the grid resolution set 0.03 for ground scene placement and 0.01 for other scenes. After voxelization, we use correlate from scipy (Virtanen et al., 2020) to find all non-penetrating positions between the object grid and the scene grid.
Then, we find the bottom surface of the object and the contact surface. The bottom surface is a 2D) grid where each cell considered part of the bottom if any one of the bottom kbottom voxels at that position is occupied in the object grid. The contact surface is defined as follows: from the bottom surface, each voxel searches below for Ksearch voxels, and if scene voxel is found, it is considered a contact. In most cases, we set Kbottom and Keorrch to to ensure stability and organization. However, for a random scene, such as placing a box full of boxes and cans, we set both values to 5 to allow for more messy and natural placement. With the contact surface, calculate convex hull that surrounds all contact surface voxels, and the placement is considered valid if the projection of the center of mass of the object falls within the convex hull.
After all placement candidates are found, we rank them based on the parameters specified in the predicates, such as the closeness to the specified relative pose or the specified support ratio (which is defined as the area of the support surface over the area of the bottom surface), and choose the one that fits the best as the object's initial placement.
After all objects are assigned an initial placement, we run a physics simulation that allows all objects to free fall. Only objects with no large displacements are regarded as successfully placed, and their poses after the simulation will be their final poses.
A.4.4 DETAILS ON STABILITY MEASUREMENT
We measure the stability of an object's placement by injecting perturbations into its physical pa- rameters and validating the balance of the batch samples via simulation. Specifically, model a configuration by a perturbation vector x = [Ap, Ar, Ac, Au, Am] E R with a nominal value To = O, where Ap E R3 is a 3D position shift, Ar E TR3 is a small-angle rotation (axis-angle) shift, Ac E R is a center-of-mass shift, Aμ E E2 is driction-coefficiene change, and Am E IR is mass change (so d = 11). Each dimension has a preset standard deviation 0 E Ra,, giving a diagonal covariance = diag .................... ......... ..). We draw N perturbations and label them via physies engine: xy (0, E). y, = 1 [fall under w,} {0, 1}. yielding D = {(5,, (1,,,,,)) W1)} For any query x (e.g.. x 0), we define the Mahalanobis distance d (z, 2²)² = (z, x)
(z. 18y-00 x) and kernel weight wy (z) = exp(- dz, the weighted failure and total counts are sz)) Σx= (z(2) y/ and n(x) = Eド= (z(x), giving the local failure probability (z)) =(x) / /=x), which gives us a sense of the stability level. In addition, we can repeatedly optimize these parameters to find highly unstable state by choosing the most-uns yet still-stable configuration among the simu- lated points at each iteration, letting 5 = {i one may then re-center at x² and repeat the process to refine.
0} and selecting x* arg maxes (2;))
A.4.5 ADDITIONAL. IMPLEMENTATION DETAILS
We use the 04 04-mini model the LLM agent for generating predicates and asset descriptions. For the spatial solver, we use Shapely y (GEOS contributors, 2024) to compute convex-hull overlaps in batch. The solver runs for 10 iterations; in each iteration, loop through each parameter and uniformly sample 40 candidate values around the current estimate. Sampling ranges depend on the scene scale: we set distance parameters to one-tenth of the shortest scene dimension and rotation parameters to 10P Since convex-hull computations can be batched, the full process completes in under seconds. For the physical solver, we use Genesis (Authors, 2024) as the physies engine.
A.4.6 EVALUATION METRICS IMPLEMENTATION
We primarily adopt three quantitative evaluation metries: VQA Score, GPT Ranking, and Settle Distance. Prior to evaluation, we re-render all scenes generated by different baselines to ensure con- sistent lighting conditions, viewing angles, and rendering parameters. For the VQA Score, we input
26

---

Published as a conference paper at ICLR 2026
the rendered scene image into GPT-4o with the prompt "Does this image show (scene-prompee Please answer with Yes or No." and measure the probability of the "Yes" token in the response, For GPT Ranking, for each prompt, we aggregate the generated examples from all baselines and feed them to GPT-4o, requesting a ranking based on scene quality and alignment with the prompt. For Settle Distance, we initialize objects generated configurations. We run the simulation for 400 steps and calculate the displacement of each object between initial and final positions. The displacement distance is clamped maximum of (to account for objects falling off the table), and we report the average across all scenes.
physics simulation with standard gravity according their
A.5 EXPERIMENT DETAILS
A.5.1 BASELINE IMPLEMENTATION DETAILS
For Architect (Wang al., 2024b), we use their official implementation together with our own 3D as- set database. 3D-Gennraliis (Sun et al., 2025b) is not open-sourced, so we re-implement their Asset- Level Policy following the instructions in the paper. The core of their pipeline is Molmo (Deitke et al., 2025), an open-source VLM that can directly output pixel coordinates. We use the same prompt as presented in their paper, and integrate Molmo with our grid-based search module to check whether the model's predicted placement is collision-free and supported. For LayoutVLM (Sun et al., 2025a), we also use their official implementation. Note that in their codebase, Layout requires text prompt paired with list of specified 3D assets input. To adapt this method to our text-only scene generation setting, we introduce an additional pipeline that queries GPT-4o to generate list of object descriptions, and then CLIP features retrieve corresponding 3D sets from the database. To ensure a fair comparison, we process our 3D asset database (also sourced from Objaverse) into the format required by LayoutVLM.
A.5.2 ANALYSIS OF ABLATION STUDY ON FEEDBACK SYSTEM
As shown in Table our full feedback system significantly outperforms Ours w/o report, demon- strating the effectiveness of explicit spatial information feedback in guiding the correction process. Moreover, Ours w/o report outperforms Ours w/o feedback, which highlights the value of the other feedback components (i.e., Grammar Feedback and Solver Failure Feedback). These provide crucial information for the LLM-Agent to adjust its actions beyond what simple binary signal can offer.
In addition, we tested visual feedback for failure recovery as an optional component. While the Ours Visual variant reduces the number of retries, it introduces additional time overhead due to API calls and rendering (note that rendering time is already excluded from the results in Table 2). Making this component optional allows our LLM-Agent to operate using only text input, which offers greater flexibility in model deployment.
A.5.3 ANALYSIS ABLATION STUDY ON PLACEMENT DESIGN
The results presented in Table 3 show that scenes generated by our framework have significantly better perceptual quality (VQA Scores and GPT Ranking) than both the Random and LLM-Only baselines. This demonstrates the importance of our predicates and solvers in creating semantically coherent arrangements.
In terms of physical plausibility (Settle Distance), the Random baseline performs well because its implementation ensures placement on a supporting surface and avoids complex stacking. The LLM-Only baseline, which lacks any explicit boundary or collision checks, performs the worst on this metric. These findings confirm that our method successfully ensures both high perceptual quality and physical stability in the generated scenes.
A.5.4 STACKING CONTROL ANALYSIS
We demonstrate controlled scene generation with varying stability levels. As shown in Fig. 15, our method can generate both stable configurations (e.g., cups stacked securely inside one another) and fragile (e.g., a cup precariously balanced top of another). This capability extends to
27

---

Published as a conference paper at ICLR 2026
stabilit 1.00
stabiliy 0.78
stabilty 0.48
stabilty - 0.07
stable stacking
unstable stacking
Figure 15: Generated placements for different stability level.
diverse objects and more complex arrangements such cans, stacks of dishes, bowls, and utensils, as illustrated in Fig. and Fig 3.
Our probabilistic programming framework, combined with Bayesian optimization, enables fine- grained control over object position, rotation, and physical properties to target more stable or un- stable states. As shown in the stacking examples, we can adjust objects' centers of mass within a feasible range to create visually implausible yet physically realizable unstable states, where more than 90% of sampled perturbations lead to collapse. This ability to intentionally generate unstable scenarios provides valuable tool for constructing challenging benchmarks and datasets for robotic manipulation.
A.5.5 ROBOT DEMONSTRATION GENERATION
We build an automatic pipeline to generate robot manipulation demonstrations for our robot exper- iment. The general automated manipulation data-generation pipeline consists of four main com- ponents: (1) Grasp sampler uniformly samples 5000 grasp candidates on the object and filters out invalid ones by lifting the object with flying gripper in physics simulation, serving processing stage for each object. It then further filters valid grasps by checking their reachability with the robot arm; Goal state generator samples goal states based on robot-arm reachability and semantic constraints. Specifically, we consider two types of semantic relations: on top of and inside. Each relation has predefined distribution over relative poses. For on top of. we sample goal poses on top of the target object's bounding box. For inside, we sample rotations such that the object's longest dimension aligns with the z-axis and allow overlapping between object bounding boxes. After sampling poses, we check validity by verifying that the robot can reach each pose with a valid grasp via inverse kinematics (the valid grasp provide relative pose between the end-effector and the object) and that the pose is penetration-fiee goal states, we let the scene settle by running a physics simulation after the gripper releases the object, and then determine whether the result is successful final state using primitive verifier function combined with a VLM-based verifier. The primitive verifier exposes functions such as bounding box queries and collision checks, and we prompt LLM to write higher-level verifier function using these primitives. For the VLM verifier, we render the initial scene, the final scene, and close-up view of the target objects the final scene (with the targets highlighted with borders) and ask the VLM whether the task has succeeded. We label a state as successful only if both veri- fiers agree; (4) Motion planner- - given the object's initial pose, the grasp pose, and the goal pose for release, we use cuRobo to plan trajectories both for reaching the grasp pose and for placing the object at the goal state after successful grasp, thereby generating the final robot trajectory.
pre-
(3) Verifier after generating several valid
28


Reviews:
[{"review_id": "OxGtXVgQqU", "reviewer": "Reviewer_LKXm", "summary": "The paper proposes a framework that leverages LLMs and a physics engine to generate physically plausible scenes. Specifically, given a library of 3D assets and a language caption, an LLM is first used to iteratively propose relevant assets and predicates that determine their initial positions. Then, spatial and physics solvers are used to ensure the layout is collision-free and physically plausible. Experiments show that the model achieves good performance compared with previous baselines, especially on scenes with cluttered objects.", "questions": "1. Many experimental details are missing:\n\n   a. How many prompts are used in the experiments, and how were they created?  \n   b. How many examples were generated per prompt?  \n   c. How exactly are the VQA score and GPT-based ranking implemented?\n\n2. From the method description and the prompt in A.3.2, it seems the proposed method does not support placing objects on other objects at a specified height (e.g., a certain shelf level). How is this achieved in Figure 5?\n\n3. The robot experiment is an interesting way to show the diversity of generated scenes from a single prompt:  \n   a. Why is the success rate of reaching much higher than that of placing?  \n   b. How are the human-designed test scenes different from the generated scenes?  \n   c. What do the generated scenes from each model look like? It would be helpful to show sample visualizations to illustrate quality and diversity.\n\n4. What are the failure cases of the model? I am especially interested in scenes with low VQA scores or GPT rankings.\n5. It would be helpful to provide qualitative examples of the ablation study between Random, LLM-Only, and the full method. What about using only the spatial solver without the physics solver?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well written and easy to follow.\n2. The generation results look good.\n3. The proposed method enables a certain level of controllability, such as the distance between objects and the stability of objects.", "weaknesses": "1. I think the term scene generation used here is misleading. The paper mostly focuses on “object arrangement” [1] or “layout generation” [2], where the goal is to place objects of similar sizes on a given surface (e.g., a bookshelf or a table). This is implied by all the qualitative examples. In contrast, scene generation usually refers to generating larger and more complex indoor scenes containing objects of various sizes and more diverse object relationships, which is not demonstrated in the experiments. I agree that the paper tackles a challenging problem involving arranging a large number of objects in confined space, but it is different from the indoor scene generation problem that the baselines address. If the authors want to keep using this term and still treat Architect / 3D-Generalist as the main baselines, they should provide more results on indoor scenes to demonstrate the effectiveness of the method.\n2. LayoutVLM [2] is an important missing baseline. This model uses a similar asset library to specify spatial relations between objects. It would be useful to compare these methods. Unlike PhyScenesis, which only leverages an LLM and a physics simulator to avoid collisions, LayoutVLM prompts a VLM to get initial object positions, which may yield more semantically meaningful results.\n3. The technical contribution is limited. SceneCraft [3] also proposes an agentic framework that generates 3D scenes using an LLM to generate Blender code with a feedback loop. Spatial and physical predicates are also used in LayoutVLM [2]. The design of the spatial and physics solvers here is largely heuristic and feels ad hoc. They seem specifically designed for cluttered scenes with small objects, which may limit applicability to other scenarios like full indoor scene generation or settings with fewer objects.\n4. I appreciate the authors’ effort in building such a complex system to achieve good results. However, it would be better to provide more experiments and details to systematically justify the design choices. See my questions below for more information.\n\nMinor Point:\nLine 106: SceneThesis also has a module for optimizing the physical plausibility of generated scenes, including collision avoidance and stability.\n\nReferences:\n1. Line 106, Scenethesis also has a module for optimizing the physical plausibility of the generated scene including collision avoidance and stability. \n\n[1] LEGO-Net: Learning Regular Rearrangements of Objects in Rooms. Qiuhong Anna Wei, et al. CVPR 2023 (Missing citation)\n\n[2] LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models. Fan-Yun Sun, et al. CVPR 2025 (Missing citation) \n\n[3] SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code. Ziniu Hu, et al. ICML 2024 (Missing citation)\n\nI am happy to raise my score if my questions and concerns are addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bZY3RF3O4a", "reviewer": "Reviewer_MLGL", "summary": "The paper addresses the task of generating physically plausible environments. To tackle challenges in both spatial arrangement and physics, the authors propose the PhyScensis framework, which leverages a large language model (LMM) to generate predicates and employs a physics engine as the physics solver. Their framework also incorporates feedback from the physics engine back to the LMM for further refinement, resulting in realistic layouts and physically stable scenes. Experimental results demonstrate superior performance compared to previous methods.", "questions": "* How are objects selected—only at the category level, or is there a more detailed retrieval?\n* Are there predefined rules for selection, or is it random? For example, in the “table for 4” case, why are all plates the same? Is this constraint imposed by the LLM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* The overall system, which integrates LLM-based predicates, a physics-based solver, a geometry-based spatial solver, and feedback to the LLM, is well-designed. This results in layouts that are both reasonable and physically stable.\n* Physics-plausible scene generation is an interesting and important direction, particularly for large-scale scene generation.\n* The experiments are thorough, including ablations and additional evaluations on downstream robotics tasks.", "weaknesses": "* It is unclear what text prompts are used in the test set for all methods. How many prompts are there, and how diverse are they?\n* There is no discussion of failure cases, particularly regarding physics. What are the limitations of the current predefined predicates?\n* Regarding the LLM, it is unclear how it determines object sizes and how it selects objects from the candidate object set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pBkHx84eJ0", "reviewer": "Reviewer_1kHh", "summary": "The paper introduces PHYSCENSIS, a framework that automatically generates interactive and physically plausible 3d environments for robotic manipulation. PHYSCENSIS leverages an LLM to propose realistic scene configurations, including spatial relations and object properties. A physics solver then checks if the proposed configuration is feasible and places the objects in the scene. If objects are not solvable, a feedback will be provided to the LLM to refine the scene configuration. Experiments show that PHYSCENSIS outperforms two baselines and that the environments can be used to train an IL policy.", "questions": "See above and\n- How does the framework compare against other frameworks in runtime?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is very well written, well motivated, and easy to follow\n- The methodology, although not entirely novel, is promising.\n- The results and ablations show that the individual design choices result in improved generation speed and scene quality.", "weaknesses": "### The main weakness of the paper is the experiments. In particular, the downstream experiment fails to showcase the advantages of the approach compared to existing scene generation pipelines in the robotics domain:\n- The VQA-based evaluation is questionable. It’s not clear if this metric works well for complex 3D tabletop environments. The high variance across models suggests it may not be reliable. Comparing it with human judgments could help validate this.\n- It’s unclear whether the same VQA model and scores are used both for evaluation and for providing feedback during generation. If so, this would bias results in favor of PHYSCENSIS, since it would directly optimize for the evaluation metric.\n- The chosen baselines are rather weak. The authors should explain why they did not compare against similar LLM + physics-based methods (e.g., ClutterGen, RoboGen, SimGen) and elaborate on the choice of baselines further\n- The downstream manipulation task is too simple and does not demonstrate the framework’s claimed advantages. The task does not depend on accurate physics or object properties. More challenging tasks like stacking, unstacking, and manipulating objects with different stability would provide stronger evidence.\n- The authors state that the cup and plate are fixed for each scene. Is this also the case for the baselines? During evaluation, are the plate and cup also fixed? If yes, the policy would not need to rely on visual cues.\n\n\nThe proposed method is not entirely novel, but it combines existing approaches. However, the problem is very relevant, and the framework could allow for training policies more robust to difficult settings in the pick and place task. However, in its current state, the experiments fail to showcase the effectiveness of the framework in that regard. Thus, in its current state, I tend towards reject. Performing extended robot experiments in more diverse environments and clarifying the evaluation methodology would strengthen the contribution and further support the paper's claims.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Jixdfnmf5M", "reviewer": "Reviewer_ummC", "summary": "The paper presents a scene generation system for complex 3d scenes that prompts an LLM to produce scene descriptions using a scene predicate domain-specific language. Candidate predicates are processed by a 2d layout constraint solver and 3d physics engine to optimize and evaluate the scenes. Feedback from the DSL generation (syntax) and solvers is used to enable the LLM to iterate toward functional scenes with physical properties like asset placement instability.\n\nExperiments test against two prior systems for 3d scene layout and ablations of some of the system components. Evaluations compare whether scenes match prompts, preference comparisons when evaluated by GPT, and the physical distances assets move after initialization (as a physical check). There is also an evaluation of model learning from demonstrations on these scenes.", "questions": "# questions\n- Table 1, 2, 3: Results should include statistical tests for differences and effect sizes. Some of the outcomes look to have overlapping standard deviations, suggesting the differences may not be large.\n- Section 4.3: How were demonstrations generated: by humans? an automated process?\n\t- The section on demonstration generation and training is very compressed and hard to follow. I was not clear on what the demonstrations were, what training was done, and how evaluation was done.\n- What costs are involved in each method evaluated (including ablations) and how do they scale?\n\t- For example: how many LLM queries used, how many iterations / computation (for solvers), how much wall clock time?\n\t- How do these costs scale with the scene size or number of assets? Other relevant input or output parameters?\n- Table 2: Why are the VQA Score, GPT Ranking, Settle Distance not included?\n- Is there any evidence around output scene diversity? How that impacts learning the outcomes?\n\t- It's often desirable that a generator can produce many different outputs from the same prompt, but this can be in tension with controllable outputs.\n\t- These metrics could be computed from the output scenes themselves, to measure things like number of assets, asset diversity across generations, placement diversity, and so on.\n\n\n# suggestions / minor comments\n- Figure 1: Why does \"more compact\" look the same?\n- How would the dock scenario shown in the figures be used for manipulation?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "# originality\nThe primary novelty is integrating multiple solver types as optimization and feedback mechanisms. This extends prior works and shows how to integrate more parts into generation and LLM guidance.\n\n# quality\nShows some promising results on learning from scene demonstrations.\n\n\n# clarity\nProvides ample qualitative examples of the method to complement quantitative results.\n\n\n# significance\nWill be of interest to the robotic manipulation community.", "weaknesses": "# originality\nNo single component of the system is particularly novel. And the use of a DSL is more constrained than the more generic code generation of the 3DGeneralist prior work. None of this is horrible, but limits novelty.\n\n# quality\nSee the questions for detailed remarks and suggestions. The primary concerns are:\n- (1) Lack of statistical testing for differences and their magnitudes.\n- (2) Need for scaling analysis to get a clearer sense of the cost-benefit trade-off of the new approach.\n- (3) Lack of clarity on the demonstration generation and training process. The results are strong, but this is marred by ambiguity on how much the task reflects a particularly strong scenario for PhyScensis compared to previous efforts (the dinner table setting task).\n\n# clarity\nSee the questions for minor comments. The demonstrations point (3) is related.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)