PAT3D: Physics-Augmented Text-to-3D Scene Generation
Authors: Guying Lin
Kemeng Huang
Michael Liu
Ruihan Gao
Hanke Chen
Lyuhao Chen
Beijia Lu
Taku Komura
Yuan Liu
Jun-Yan Zhu
Minchen Li
Venue: ICLR 2026 Poster (2026)
Keywords: Text-to-3D Generation
Vision Language Models
Rigid Body Contact Simulation
Simulation-In-The-Loop Optimization
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: We introduce PAT3D, the first physics-augmented text-to-3D scene generation framework that integrates vision–language models with physics-based simulation to produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then converted into initial simulation conditions. A rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we introduce a simulation-in-the-loop optimization procedure that guarantees physical stability and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior approaches in physical plausibility, semantic accuracy, and visual quality. Beyond high-quality generation, PAT3D uniquely enables simulation-ready 3D scenes for downstream tasks such as scene editing and robotic manipulation. Our code and data are available at https://github.com/Simulation-Intelligence/PAT3D.

Published as a conference paper at ICLR 2026
PAT3D: PHYSICS-AUGMENTE TEXT-TO-3D SCENE GENERATION
Guying Lin'
Kemeng Huang?. Michael Liu'
Ruihan Gao
Hanke Chen'
Lyuhao Chen'
Beijia Lu"
Taku Komura² Yuan Liu
Jun-Yan Zhu'
Minchen Li
CCamegic Mellon University "The University of Hong Kong
'The Hong Kong University of Science and Technology *Genesis Al
{guyingl, mliu6, ruihang, lyuhaoc, bei jialll cmu. kmluang@connect.. hk, {hankec, junyanz ) écs. cmu, edu, takuêcs, hku.hk, yuanly@ust, .k, mincheral@gmail.com
Baran bider
Scene Editing
haklnn pondst
jasl"
deciaion
mevole vidds salin paiin
Robot Manipulation
Physically implausible Seenes
Our results
Supponed Applications
Figure I: PAT3D is the first text-to-3D scene generation framework that produces simulation-ready and intersection-fre results. The left column shows results from direct depth-based arrangements, which suffer from object interpenetrations (top) and collapse under simulation due to inconsistent layouts (bottom). The middle column presents PAT3D results, where physically valid layouts re- main stable under simulation. These high-quality scenes are immediately usable for downstream applications, including scene editing and robotic manipulation (right).
ABSTRACT
We introduce PAT3D, the first physics-augmented text-to-3D generation framework that integrates vision-language models with physics-based simula- tion produce physically plausible, simulation-ready, and intersection-free 3D scenes. Given a text prompt, PAT3D generates 3D objects, infers their spatial relations, and organizes them into a hierarchical scene tree, which is then con- verted into initial conditions for simulation. A differentiable rigid-body simulator ensures realistic object interactions under gravity, driving the scene toward static equilibrium without interpenetrations. To further enhance scene quality, we intro- duce simulation ietillatiooni--ee--ooooo in-the-loop optimization procedure that guarantees physical sta- bility and non-intersection, while improving semantic consistency with the input prompt. Experiments demonstrate that PAT3D substantially outperforms prior ap- proaches in physical plausibility, semantic consistency, and visual quality. Beyond high-quality generation, PAT3D uniquely enables conulation-readyy 3D scenes for downstream tasks such editing and robotic manipulation.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
The ability to generate realistic and editable 3D scenes from natural language has broad applications across a variety of domain including virtual reality, robotics, digital twins, and content creation. Recent advances in diffusion and autoregressive generative models have significantly pushed the boundaries of text-to-3D scene generation, making it possible to synthesize high-quality object ge- ometry and compelling visual content Lin et al. (2023); Metzer et al. (2023); Michel et al. (2022); Poole et al. (2023); Chen et al. (2025b; 2024a); Huang et al. (2024); Gao et al. (2024). However, despite these advances, existing approaches struggle to ensure that generated scenes exhibit physical plausibility- critical requirement for downstream applications that demand interaction, simulation, or building a real-world correspondence.
In particular, current 3D scene generation pipelines Huang et al. (2024); Gao et al. (2024) often treat layout composition as a purely geometric problem, omitting physical reasoning entirely or using simple heuristics to prevent unfavored physical interaction such as object intersection. Due to the lack of explicit constraints from physics, this leads to common issues such as floating, unstable stacking, and incorrect support relations, ultimately limiting scene realism and usability. Earlier efforts have incorporated physical constraints to enhance single-object stability Guo al. (2024); Chen et al. (2024c), or used video diffusion priors for plausible dynamics Zhang et al. (2024), but none of these methods address the complex spatial dependencies and contact interactions required for stable and semantically coherent multi-object scenes.
One promising direction is to integrate physics- based simulation into the scene generation process to enhance physical realism. However, this approach introduces several challenges. First, objects must be represented as individually segmented 3D meshes to enable simulation of interactions under gravity and contact forces. Applying simulation to scenes represented by single connected mesh is ineffective, as fails to capture interactions between objects. Second, physics- based simulation requires well-posed initial configuration, typically free of intersections, avoid numerical insta- bility and unrealistic behavior Li et al. (2020). Yet, identifying such an intersection-free starting state is nontrivial. Finally, even if the simulated scene is physically plausible, may diverge from the intended semantics described in the input text, due to the multiplicity of valid static equilibria.
To address these challenges, we propose PAT3D, plusint-augmenteed text-to-3D scene generation framework that integrates differentiable rigid-body contact simulation into the generation pipeline. Given a text prompt, we first synthesize a reference image to reflect the spatial relations among objects. Individual objects are then generated and coarsely positioned using vision foundation mod- els Bochkovski et al. (2025); Kirillov et al. (2023b); Hunyuan3D (2025). Next, a vision-language model (VLM) Hurst et al. (2024) extracts the physical dependencies between objects from the ref- erence image, which are then organized into a scene tree. PAT3D then produces an intersection -ree initial configuration from the coarsely positioned 3D scene and scene tree through physics-guide refinement. This initialization deliberately introduces small gaps along the gravity direction for ob- jects with parent-child relations in the scene hierarchy, simplifying intersection avoidance while preserving inferred spatial relations. These gaps are later resolved through simulation, allowing ob- jects settle naturally under gravity and contact forces while making slight, physically plausible adjustments to their spatial relations. Finally, differentiable simulation is applied to further optimize the layout, improving semantic consistency in the resulting scene.
We validate our method on diverse, contact-rich scenes and demonstrate its effectiveness against existing state-of-the-art 3D scene generation approaches through both qualitative and quantitative evaluations under visual quality and physical plausibility metrics. We further demonstrate that our generated scenes are readily editable and interactable through simulation, enabling physically plausible scene editing and direct construction of simulation environments for policy evaluation in robotic manipulation tasks. Our code and data are available at https:/ //t:://gaa ////tt/tii// tig.. com/ Simulatioon
In summary, our main contributions include:
We introduce PAT3D, the first physics- augmentd text-to-3D scene generation framework that integrates vision- languaee models with physics- based simulation, achieving state-of- the-art physical plausibility, semantic consistency, and visual quality.
2

---

Published as a conference paper at ICLR 2026
We propose a physics-aware scene initialization module to prepare scenes for simulation. This module infers physical dependencies among objects, organizes them into a hierar- chical scene tree, and converts the scene tree into intersection- initial conditions for simulation.
We develop a layout optimization strategy based on artificially time-stepped differentiable simulation, enabling efficient evaluation and differentiation of static equilibrium w.r.t initial layout.
RELATED WORK
Single Object Generation. Building on the success of text-to-image generation models Rombach et al. (2022); Ramesh et al. (2022); Kang et al. (2023); Yu et al. (2022), there has been rapid progress in 3D generative models conditioned on text or images. prominent class of methods leverages 2D diffusion priors for 3D generation Poole et al. (2023); Wang et al. (2023); Lin et al. (2023); Chen et al. (2023); Metzer et al. (2023); Wang et al. (2024); Sun et al. (2024); Long et al. (2024); Michel et al. (2022), with DreamFusion Poole et al. (2023) introducing Score Distillation Sampling (SDS) to optimize 3D representations using gradients from 2D diffusion models. Subsequent works have extended SDS with multi- diffusion models, improving both 3D generation quality and single- view reconstruction Liu et al. (2023a); Wang & Shi (2023); Shi et al. (2024); Liu et al. (2024b); Zhou & Tulsiani (2023); Liu et al. (2024a); Long et al. (2024); Shi et al. (2023); Liu et al. (2024d). Another research direction trains large-scale transformers to generate 3D shapes in a ford-forward manner Hong al. (2024); Li al. (2024); Xu al. (2024b); Tochilkin et al. (2024), relying on curated, large-scale 3D asset datasets. While these models can generate visually compelling shapes, they often ignore the physical properties, such as stability, of the object, which are essential for real-world applications. To address this. recent efforts have incorporated physics- simulation into the generation pipeline to produce self- supportup 3D objects by optimizing physical attributes such mass distribution Guo et al. (2024); Chen et al. (2024c); Yan et al. (2024); Cai et al. (2024). Additionally, PhysDreamer Zhang et al. (2024), optimizes physical properties like Young's 's modulus and initial velocity to generate dynamic motions that are both visually plausible and physically grounded, guided by video diffusion priors.
Scene Generation. While single-object generation methods produce visually appealing assets, they often lack scale awareness and spatial grounding, making scene composition challenging. The primary bottlenecks in 3D scene generation include decomposing scenes into individual assets, esti- mating their relative scale and pose, and ensuring physical feasibility (e.g., contact, stability). Sev- eral works address these challenges through multi-stage pipelines. Early methods such as Vilesov et al. (2023); Chen et al. (2024b); Han et al. (2024) adopt object-centric reconstruction followed by layout and geometry optimization using physical constraints or differentiable rendering. Shriram et al. (2024) lifts the scene image to 3D point clouds as whole, inpaints occluded regions, and refines appearance using 2D diffusion priors. Recently, Large Language Models (LLMs) and VLMs are increasingly leveraged to infer spatial relations and scene structure. Gao et al. (2024) constructs a scene graph with objects as nodes and their relations as edges. Zhou et al. (2024) uses a VLM to generate coarse layout, which is subsequently refined with rendering losses and physical con- straints. Yao et al. (2025) infers a scene graph describing simplified pairwise relationships between objects, and use them to optimize object poses and scales. Wang et al. (2025) similarly leverages LLM-based reasoning to query objects' relative sizes and physical properties, enabling more plausi- ble scene layouts. However, none of these methods can ensure physically accurate contact handling or maintain physical stability in the generated scene. Scene-level optimization under SDS loss is commonly used for joint geometry-text alignment Zhou et al. (2025b), while Huang al. (2024) and Xu et al. (2024a) explore multi-instance and 4D compositional generation guided by spatial and trajectory priors. Building on similar SDS-based refinement, Zhou et al. (2025a) generates composi- tional 3D scenes from 2D layouts, enabling controllable instance placement but still lacking accurate modeling of physical interactions between objects. Other approaches incorporate physical property estimation into 3D representations, either from visual cues Zhao et al. (2024) or explicit user in- put Chen et al. (2025a), to support dynamic simulation or interaction Liu et al. (2024c). Recent systems such Blender-MCP, Huang et al. (2025b), and Li et al. (2025) integrate LLM reasoning into graphics tools and simulation engines, enabling interactive behavior and fine- -rainer control. Sun et al. (2025) propose LayoutVL.M, which uses VLMs to generate differentiable spatial rela-
3

---

Published as a conference paper at ICLR 2026
onjects and spattai Rebitton Extrattion
Layout intialaatinn
Logostt Optimbation
No Leeud
Adbe
New
Overview of our text-to-3D scene generation pipeline. (a) Given input text, a ref- erence image is first generated to capture spatial relations among objects, from which 3D assets are generated using vision foundation models, and scene tree is extracted using VLM. (b) Assets arranged into an initial layout using 3D priors from monocular depth estimation (left), then refined with the scene tree to produce intersection-free configuration for simulation (right). (c) For- ward simulation ensures physical plausibility but may distort semantics (left). We address this with simulation- -n-thh---ooo optimization, enforcing semantic consistency and physical validity (right).
Figure 2:
tions and jointly optimize 3D layouts in indoor scenes. However, most existing scene generation methods focus primarily on layout composition. They either omit physical reasoning altogether or incorporate only simple physics priors to prevent object interpenetration, without modeling accurate contact interactions or ensuring physically stable scene layouts. We thus address this gap by novelly augmenting text-to-3D scene generation with differentiable rigid body contact simulation.
METHOD
Our framework comprises three stages: 3D object and spatial relation extraction (subsection 3.1), where 3D assets are generated from text and its spatial relation are organized into scene tree; layout initialization (subsection 3.2), which first arranges generated assets using monocular depth priors obtained from refernece image and uses scene tree to refine them into an intersection-free configuration; and layout optimization (subsection 3.3), where a cowuulaiionii-the-looe optimization procedure is applied to ensure physical plausibility and improve semantic consistency of 3D scene.
3.1 3D OBJECT AND SPATIAL RELATION EXTRACTION
Since directly producing both 3D objects and layouts with text-to-3D models and LLMs often fails to capture complex spatial relations, instead employ text-to-image model generate reference image that guides object generation and scene tree construction. See Figure 2(a).
3.1.1
3D OBJECTS GENERATION
To generate individual objects for the scene specified by the text prompt, a VLM is queried with the reference image to obtain object class labels, and the image is segmented with Grounded-SAM Kir- illov et al. (2023a); Ren et al. (2024); Liu et al. (2023b) accordingly. Based on the segmented object regions, further prompt the VLM to generate detailed text descriptions encompassing object mantics, material, color, and orientation. These descriptions are fed into a text-to-3D pipeline Hun- yuan3l) (2025) synthesize high-quality, textured 3DD assets that are both semantically consistent and visually realistic.
3.1.2 SPATIAL RELATION EXTRACTION
We then extract the relative spatial relations among objects in the scene from the reference image and analyze their physical dependencies. This information provides essential guidance for intersection- free layout initialization (subsection 3.2) and subsequent optimization (subsection 3.3). Specifically, for each pair of segmented objects that appear with similar horizontal positions and adjacent vertical positions the reference image, we prompt a VLM to infer their dependency along the gravity axis, identifying relations such as on, contain, and support. The resulting pairwise relations are then
4

---

Published as a conference paper at ICLR 2026
organized into hierarchical scene tree that encodes how objects support one another under gravity. Starting with the ground as the root node, we traverse the scene and iteratively add objects as nodes in the tree. For each unvisited object that has direct physical dependency with existing node, insert it as a child of that node. This recursive process continues until all objects have been included. Additional details are provided in Algorithm 1, and an example is shown in Figure 2(a).
3.2 LAYOUT INITIALIZATION
To obtain an intersection-free and semantically consistent initial layout for the subsequent iitlatiionni-the-lloe optimization, we first compute the translational and scaling transformations to build preliminary layout consistent with the reference image, and then refine it using the ex- tracted scene tree ensure no object intersection and stronger physical constraints. See Figure 2(b).
3.2.1 PRELIMINARY LAYOUT
Our straightforward approach arranging the objects generated in subsubsection 3.1.1 into layout consistent with the reference image is to back-project the 2D reference image with depth estima- tion to obtain each object's 3D point cloud. Scaling and translational transformations computed by aligning the object's center with the centroid of its point cloud. In practice, however, heavy occlusions in the 2D image make scaling unreliable when derived directly from partial point clouds. To address this, we first query the VL. to identify the least occluded object in the scene and it compute relative scaling for the other objects by prompting the VLM to inpaint occluded regions of the 2D image and estimating scaling factors from the bounding boxes of the inpainted objects. Each object's 's final transformation is obtained by combining the global and relative scaling factors, followed by alignment with the projected 3D point cloud. This procedure produces the preliminary layout shown in Figure 2(b).
then be
anchor compute global scaling transformation for the entire scene. We then
3.2.2 REFINED INITIAL LAYOUT
To refine the layout under physical dependency constraints and ensure non- non-intersection, we traverse the scene tree in breadth-firy first manner and apply horizontal and vertical refinements each node.
Horizontal refinement. We enforce two rules: (1) Parent-child: the projection of the child must lic entirely within that of the parent (e.g., fruits inside basket); (2) Sibling: objects sharing the same parent must have non-overlapping projections (e.g., a vase, plate, and fork on a table).
Vertical refinement. Each child is lifted above the bounding box of its parent along the gravity axis, preventing intersections.
This simple strategy, compared with more complex optimization methods, efficiently resolves inter- sections while preserving semantic constraints, providing favorable initial conditions for simulation. The refined results are shown in Figure 2(b).
3.3 LAYOUT OPTIMIZATION
After simulation, gravity causes child objects to fall onto or into their respective parents, and sibling objects naturally adopt physically plausible poses. However, due complex inter-object interac- tions, simulation alone may cause the scene to deviate from its intended semantics. To address this, we introduce simulation-in -oott-loo optimization to improve semantic consistency in the simu- lated scene. See Figure 2(c).
Specifically, we refine our intersection free initialization qu so that the final equilibrium state gx+1 better aligns with the scene tree:
min L(gn+z(gu)) s.t. f(gn+1) = 0,
(1)
where L measures semantic inconsistency and f denotes the net force on all objects..
For each object i with container I, we define its projected bounding box on the horizontal plane as BBox; (p }. }. The local loss penalizes deviations of the corners of from BBox,:
4 d(pinnn 2 + d(pinax: BBox,)
5

---

Published as a conference paper at ICLR 2026
where d(p, BBox) 0 if p BBox, otherwise, d is the Euclidean distance from p to the box boundary. The total loss defined as
N
A(gx+1990))
4
(3)
where N is the total number of objects in the scene.
Direct gradients of (++1 with respect to Yo cannot be obtained by differentiating the static equi- librium constraint f(9n+1) 0, since go serves only as the initial guess and is not part of the constraint. Differentiating through the nonlinear solver also prohibitively expensive. Instead, we adopt artificial time-stepping formulation Fang et al. (2021), in which the quasi-static system gradually evolves toward equilibrium across intermediate states. This enables efficient backpropa- gation from gn++1 i to do via implicit differentiation each step. See Appendix B for more details our forward simulation method and the derivation of differentiation.
4 EXPERIMENTAL RESULTS
graee bie-hotdded drsk bands containr two here of white and bage
OBM
oooo
Fow stacked cupt and four
stocked plotse
frote, there
On table, ohere a vise with éouqued Rowers Basile it, there plate of oake-
A vistoge wooden radio with o small ane figurine on cap whtr on stecé of clrne hardcover books, next to weeden tup holdiny colorfal potcils
Text prompts
(a) GraphDreamer
(b) Blender-MCP
(c) MIDI
(d) Ours
Figure 3: Comparison to baseline methods. The scenes are generated from our text prompts.
OOM indicates out of memory.
4.1 COMPARISON
4.1.1 BASELINES
We compare our method against three baselines: GraphDreamer Gao al. (2024), MIDI Huang et al. (2024), and Blender-MCP or ahujasid (2025). Both GraphDreamer and Blender-MCP take text prompts as input, while MIDI uses reference image input. To ensure fair comparison, we provide MIDI with our scene reference image their input.
4.1.2 DATASET
Since there is no standard benchmark for general scene generation, construct test dataset consisting 18 text text prompts. Among them, 3 prompts are taken from MIDI, and 2 prompts are from GraphDreamer. Additionally,
LIM to generate 13 new text prompts spanning diverse

---

Published as a conference paper at ICLR 2026
Clip Score f
VQA Score f Displacement
Pene. Ratio
Phys. Score
GraphDreamer
27.53
0.46
0.25
61.72
40.0
Blender-MCP
28.93
0.56
1.03
14.78
47.7
MIDI
29.68
0.63
0.69
110.80
62.7
Raw layout
29.88
0.64
0.81
14.11
65.5
Scene Init.
30.77
0.70
2.91
34.2
Ours
31.79
0.68
0
88.5
Table 1: Quantitative Evaluation. Our method achieves the highest semantic consistency with input text prompts among all baselines, and is the only method that achieves perfect physical stability and non-intersection. We also ablates results without layout initialization and optimization, shown as raw layout.
scenes. These prompts describe physical interactions between objects, including stack books and basket of fruits. Additional 3D results generated by our method, in comparison with the base- lines, along with corresponding text prompts and reference images, can be found on visualization website'. Beyond these comparisons, we also present 12 more examples produced by our method Appendix
4.1.3 EVALUATION METRICS
We evaluate our generated scenes using five metries: CLIP ScoreRadford al. (2021), VQA ScoreLin et al. (2024), Simulated Scene Displacement (D), the Ratio of Penetrating Triangle Pairs (R), and Physical Plausibility Score. Together, these metrics measure semantic consistency, physi- cal stability, interpenetration, and overall physical plausibility. Details of all metrics are provided in Appendix F.
4.1.4 PERFORMANCE AND DISCUSSIONS
In Figure 3, compare PAT3D with bascline methods on five general scenes involving complex object interactions. Additional comparisons with four scenes highlighted in the MIDI and Graph- Dreamer provided in Appendix E. Importantly, in PAT3D, reference image serves only extract the complex spatial relations implied in the text prompt; the final scene does not need to remain vi- sually consistent with the reference image.
GraphDreamer struggles to scale to larger scenes because jointly optimizes both object geometry and scene layout through Score Distillation Sampling (SDS) Poole et al. (2023), which is highly resource-intensive. Moreover, GraphDreamer exhibits weak understanding of spatial relations in text prompts. As shown in the second, fourth, and fifth scenes of Figure 3, it often ignores spatial constraints, leading to chaotic object arrangements. Blender-MCP generates layouts with little phys- ical realism. the first and second scenes of Figure 3, the razor and toothbrush float above the cup, and the plate is suspended in mid-air. It also produces objects with unrealistic scales: in the fourth scene, the cake and vase appear disproportionately small compared to the table. MIDI encounters difficulties when handling scenes with complex object contact, seen in the first, second, and fifth scenes of Figure 3, objects often appear in irregular yet tightly packed configurations. Although interpenetration is avoided, the resulting layouts cluttered potentially because MIDI generates the entire scene a single step, the quality of individual objects is compromised.
By contrast, method decomposes the scene generation process, iteratively creating objects to ensure high-quality results. Leveraging VL.M-based guidance together with physics simulations, PAT3D produces 3D scenes that are both physically realistic and semantically consistent, even in scenarios with complex object interactions. Quantitative comparisons are shown in Table 1. Com- pared to baseline methods, which frequently suffer from object intersection and floating artifacts that undermine physical plausibility, our approach consistently produces stable, penetration- free ar- rangements. Our method also achieves the highest semantic consistency with the input text prompts.
Pittps://3das http://ddimmmb/e/ babeel ieeeeiiieee visuall izat in.iion... onn.ennn wt..ifyy.pp............. app/

---

Published as a conference paper at ICLR 2026
4.2 APPLICATION
Our generated simulation- ready scenes can be directly imported into a simulator for downstream applications. We demonstrate two such applications: scene editing and robotic manipulation.
4.2.1 SCENE EDITING
We demonstrate a scene editing application enabled by framework, which supports interactive manipulation while preserving the physical plausibility of the entire scene, including object addition and deletion. By leveraging our physics-based simulation backend, the edited scene converges to a force-equilibrim state without mesh intersections. Figure highlights an example showcasing object addition and deletion, and animation results are provided in the supplementary material.
(b)
(c)
(d)
Figure 4: Scene editing. We demonstrate the equilibrium state after addition and deletion opera-
tions: (a) initial scene, (b) deleting a book at the bottom, (c) deleting the pen holder, (d) adding
book on top.
4.2.2 ROBOTIC MANIPULATION
Our generated scenes can be directly imported into a simulator to validate robotic manipulation policies. In Figure 5, we present two illustrative examples, a failed grasp and a successful grasp, with video sequences provided in the supplementary material. Robotic manipulation applications impose unique re- quirements on scene generation: objects must be con- sistently positioned and free of interpenetrations. Our framework satisfies these requirements, ensuring that the generated scenes are well-suited for reliable pol- icy evaluation.
Successful grasp
Failed grasp
Figure 5: Policy evaluation for robotic manipulation. Example of successful and a failed grasp where the attempted action causes objects to topple.
4.3 ABLATION STUDY
We first qualitatively illustrate the impact of our layout initialization module and simulation-in-the. loop optimization module. As shown in Figure 6(a), while the spatial relations between objects extracted directly from the depth map are generally reasonable, the scene still suffers from signif- icant interpenetration: books intersecting with one another and the pen protruding its holder. By contrast, after applying our proposed layout initialization based on the scene tree, we obtain an intersection-fiee layout shown in Figure 6(b), where projections of each object along gravity direc- tion typically lies in the projections of their containers or supporters, thereby satisfying physical dependency constraints along the gravity direction.
Nevertheless, simply enforcing physical dependencies before simulation does not ensure that the resulting simulated scene would still satisfy the intended semantics. In Figure 7(a), a stack of irreg- ular blocks collapses after simulation due to unbalanced center of mass. In contrast, Figure 7(b) demonstrates that, by further optimizing the initial layout through our simulation-in-the-lee timization, the simulated scene converges to stable configuration of stacked blocks that better reflects the semantics.
op-
We further quantitatively evaluate the effecttivneess of our method by computing semantic and phys- ical metrics on both the depth-aligned layout without our layout initialization and optimization (de- noted as layout), the layout from our scene initialization module (denoted Scene Inil.), and
8

---

Published as a conference paper at ICLR 2026
(a) Before
(b) After
(a) Depth prediction
(b) Scene tree
Figure 6: Layout initialization w/o and w/ scene tree. Layouts obtained from depth pre- diction (a) without and (b) with adjustment based on the scene tree. (Text prompt: "...a neatly stacked pile of three books...". See Ap- pendix D for the complete prompt.)
Figure 7: Layout optimization. Simulated
layouts from initial layout (a) without and (b) with further optimization using differentiable simulation. (Text prompt: "a stack of colorful wooden blocks...". See Appendix D for the complete prompt.)
compare them with final output in Table 1. Our layout initialization removes penetration by sac- rificing physical stability, but enables applying our layout optimization to consistently improve all metrics. The gains in semantic consistency metries relatively smaller, they primarily depend on visual appearance factors such as geometry and texture.
5 CONCLUSION
We presented PAT3D, physics-augmente framework for text-to-3I scene generation that integrates vision-language reasoning with dif- ferentiable rigid body simulation. By de- composing the generation process into inter- pretable stages object and relation extrac- tion, layout initialization, and chysics-guides layout optimization our method produces 3D scenes that
only semantically meaning- ful but also physically plausible and simulation- ready. Through extensive experiments on di- verse, contact-rich scenes, we demonstrated that PAT3D achieves superior physical realism compared to existing approaches. We believe PAT3D represents a step forward bridging high-level scene understanding with low-level physical reasoning. We hope this work inspires further research in physically grounded, con- trollable, and editable 3D scene generation.
Figure 8: Failure case Figure 9: Failure case
2.
"A swing hanging from "A brown leather sofa a tree."
decorated with plush toys,
Limitations and Future Work We present two representative failure cases in Figure and Fig- ure 9. While PAT3D handles most common physical dependencies, certain subtle relations remain challenging. In Figure the concept of "hanging" in the prompt "A swing hanging from tree" misinterpreted, whereas physically correct configuration would require the swing to be suspended from two specific attachment points. Extending our system to a broader set of spatial relations and larger-scale scenes are both promising directions for future work. One potential avenue is in- corporate path planning techniques and adopt hierarchical optimization strategy to manage spatial complexity more effectively.
Our simulation- inntiio--o--th--ooppp reliably decreases, it does not guarantee convergence the global optimum corresponding perfect
the-loop optimization is solved using a local optimizer. Although the objective

---

Published as a conference paper at ICLR 2026
semantic alignment. This limitation appears in the failure case of Figure 9, where all stuffed toys should ideally rest on the sofa according to the prompt. However, due to the highly crowded con- figuration, the optimizer provides suboptimal outcome: it reduces the number of fallen toys from three to one, yet one gray toy still remains on the floor. As the first work to incorporate differentiable simulation into 3D scene generation, view the exploration of global optimization strategies an exciting direction for future improvements, further expanding the possibilities opened by PAT3D.
6 ACKNOWLEDGMENTS
Minchen Li was supported in part by the Junior Faculty Startup Fund of Camegie Mellon University and a gift from Genesis AL. Jun-Yan Research Grant. Taku Komura was supported by the Innovation and Technology Commission of the HKSAR Government under the I'SS--Plaffomm grant (Ref: ITS/333/23FP).
Yan Zhu was supported by the Packard Foundation and a Cisco
The authors would like to thank Yuezhi Yang. Yuanhao Wang, Lingting Zhu, Donglai Xiang, Kangle Deng, Liwen Wu, Yang Zheng. Yehonathan Lipman, Gaurav Parmar, Maxwell Jones, and Kevin You for their insightful discussions and feedback. We also extend our gratitude Xinyu Lu for his professional support and key contributions to the development of Libuipe.
REFERENCES
Aleksei Bochkovskii, Amaäl Delaunoy, Hugo Germain, Marcel Santos, Yichao Zhou, Stephan R. Richter, and Vladlen Koltun. Depth pro: Sharp monocular metric depth in less than a second. In International Conference on Learning Representations, 2025. URL https:///axii/
abs/2410..22 02073.
Junhao Cai, Yuji Yang, Weihao Yuan, Yisheng He, Zilong Dong, Liefeng Bo, Hui Cheng, and Qifeng Chen. couusiann-nnfomee continuum for physical property identification and simulation. arXiv e-prints, pp. arXiv-2406, 2024.
Boyuan Chen, Hanxiao Jiang, Shaowei Liu, Saurabh Gupta, Yunzhu Li, Hao Zhao, and Shenlong Wang. Physgen3d: Crafting miniature interactive world from single image. arXiv preprint arXiv:2503.20746, 2025a.
Jinnan Chen, Lingting Zhu, Zeyu Hu, Shengju Qian, Yugang Chen, Xin Wang, and Gim Hee Lee. Mar-3d: Progressive masked auto-regressor for high-resolution 3d generation. arXiv preprint arXiv:2503.20519, 2025b. 2025b..
Rui Chen, Yongwei Chen, Ningxin Jiao, and Kui Jia. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. IEEE International Conference on Computer Vision (ICCV), 2023.
Yongwei Chen, Yushi Lan, Shangchen Zhou, Tengfei Wang, and XIngang Pan. Sar3d: Au- toregressive 3d object generation and understanding via multi-scale 3d vqvae. arXiv preprint arXiv:2411.16856, 2024a.
Yongwei Chen, Tengfei Wang, Tong Wu, Xingang Pan, Kui Jia, and Ziwei Liu. Comboverse: Com- positional 3d assets creation using spatially-awar diffusion guidance. In European Conference Computer Vision, pp. 128-146. Springer, 2024b.
Yunuo Chen, Tianyi Xie, Zeshun Zong Xuan Li, Feng Gao, Yin Yang, Ying Nian Wu, and Chen- fanfu Jiang. Atlas3d: Physically constrained supposupn text-to-3d for simulation and fabri- cation. arXiv preprint arXiv:240S. 18515, 2024c.
Yu Fang, Minchen Li, Chenfanfu Jiang, and Danny M Kaufman. Guaranteed globally injective 3d deformation processing. ACM Transactions on Graphics, 40(4), 2021.
Gege Gao, Weiyang Liu, Anpei Chen, Andreas Geiger, and Bernhard Schölkopf. Graphdreamer: Compositional 3d scene synthesis from scene graphs. In Proceedings of the IEEE/CVF Confer- on Computer Vision and Pattern Recognition, pp. 11199-21300 2024.
10

---

Published as a conference paper at ICLR 2026
Minghao Guo, Bohan Wang, Pingchuan Ma, Tianyuan Zhang, Crystal Owens, Chuang Gan, Josh Tenenbaum, Kaiming He, and Wojciech Matusik. Physically compatible 3d object modeling from single image. Advances in Neural Information Processing Systems, 37: 19111-999692228 2024.
Haonan Han, Rui Yang. Huan Liao, Jiankai Xing, Zunman Xu, Xiaoming Yu, Junwei Zha, Xiu Li, and Wanhua Li. Reparo: Compositional 3d assets generation with differentiable 3d layout alignment. arXiv preprint arXiv:2405. 18525, 2024.
Yicong Hong, Kai Zhang, Jiuxiang Gu, Sai Bi, Yang Zhou, Difan Liu, Feng Liu, Kalyan Sunkavalli, Trung Bui, and Hao Tan. Lrm: Large reconstruction model for single image to 3d. International Conference Learning Representations (ICLR), 2024.
Yixin Hu, Tesco Schneider, Bolun Wang. Denis Zorin, and Daniele Panozzo. Fast tetrahe- dral meshing in the wild. ACM Trans. Graph., 39(4), July 2020. ISSN 0730-0301. doi: 10.1 00.115/338666..399333 URL https: http//i..ogggl/// 1ttp://0...1113338633333333
Kemeng Huang, Xinyu Lu, Huancheng Lin, Taku Komura, and Minchen Li. Stiffgipe: Advanc- ing gpu ipe for stiff affine-defommabee simulation, 2025a. URL 2411.0 06244.
Zchuan Huang, Yuan-Chen Guo, Xingqiao An, Yunhan Yang, Yangguang Li, Zi-Xin Zou, Ding Liang, Xihui Liu, Yan-Pei Cao, and Lu Sheng. Midi: Multi-instance diffusion for single image to 3d scene generation. arXiv preprint arXiv:2412.0355.. 2024.
Zhening Huang, Xiaoyang Wu, Fangcheng Zhong, Hengshuang Zhao, Matthias Nießner, and Joan Lasenby. Litereality: Graphics-ready 3d scene reconstruction from rgb-d scans, 2025b. URL
ptttss///rriien www.aiiiirr...eoeooooooooooooeee abrg//bb52507 22000///20..011.. 02861.
Tencent Hunyuan3D. Hunyuan3d 2.0: Scaling diffusion models for high resolution textured 3d
assets generation, 2025.
Aaron Hurst, Adam Lerer, Adam Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Minguk Kang, Jun- Yan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Diederik Kingma. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6960 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment anything. arXiv:2304.0264 2023a.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF international conference computer vision, pp. 4015-4026, 2023b.
Lei Lan, Danny M. Kaufman, Minchen Li, Chenfanfu Jiang, and Yin Yang. Affine body dynamics: fast, stable and intersection-free simulation of stiff materials. ACM Trans. Graph., 41(4), July 2022. ISSN 0730-0301.
Jiahao Li, Hao Tan, Kai Zhang, Zexiang Xu, Fujun Luan, Yinghao Xu, Yicong Hong, Kalyan Sunkavalli, Greg Shakhnarovich, and Sai Bi. Instant3d: Fast text-to-3d with sparse-view gen- eration and large reconstruction model. In International Conference on Learning Representations (ICLR), 2024.
Minchen Li, Zachary Ferguson, Teseo Schneider, Timothy R Langlois, Denis Zorin, Daniele Panozzo, Chenfanfu Jiang, and Danny M Kaufman. Incremental potential contact: intersection- and inversion-free, large-deformation dynamics. ACM Trans. Graph., 49,4:449 2020.
11

---

Published as a conference paper at ICLR 2026
Qixuan Li, Chao Wang. Zongjin He, and Yan Peng. Phip-g: Physics-guides text-to-3d compositional scene generation. arXiv preprint arXiv:2552.00770 2025.
Chen-Hsuan Lin, Jun Gao, Luming Tang, Towaki Takikawa, Xiaohui Zeng, Xun Huang, Karsten Kreis, Sanja Fidler, Ming- Yu Liu, and Tsung- Yi Lin. Magic3d: High-resolution text-to-344 content creation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2023.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation, 2024. URL ht https: : /awpiviivvv//.////////.///aaaav. org/ a 4404.4444 01291.
Minghua Liu, Ruoxi Shi, Linghao Chen, Zhuoyang Zhang. Chao Xu, Xinyue Wei, Hansheng Chen, Chong Zeng, Jiayuan Gu, and Hao Su. One-2-3-45++: Fast single image to objects with consistent multi-view generation and 3d diffusion. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2024a.
Minghua Liu, Chao Xu, Haian Jin, Linghao Chen, Mukund Varma T, Zexiang Xu, and Hao Su. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In Conference Neural Information Processing Systems (NeurIPS), 2024b.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; Zero-shot image to 3d object. In IEEE International Conference on Computer Vision (ICCV), 2023a.
Shaowei Liu, Zhongzheng Ren, Saurabh Gupta, and Shenlong Wang. Physgen: Rigid-body physics- grounded image-to-videe generation. In European Conference on Computer Vision, pp. 360-378. Springer, 2024c.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Chunyuan Li, Jianwei Yang, Hang Su, Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. arXiv preprint arXiv:2303.05499 05099. 2023b.
Yuan Liu, Cheng Lin, Zijiao Zeng, Xiaoxiao Long, Lingjie Liu, Taku Komura, and Wenping Wang- Synedreamer: Generating multiview
weattiiew--consistee consistent images from a single-view image. In Interna- tional Conference on Learning Representations (ICLR), 2024d.
Xiaoxiao Long, Yuan-Chen Guo, Cheng Lin, Yuan Liu, Zhiyang Dou, Lingjie Liu, Yuexin Ma, Song-Hai Zhang, Mare Habermann, Christian Theobalt, et al. Wonder3d: Single image to 3d using cross-domain diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Gal Metzer, Elad Richardson, Or Patashnik, Raja Giryes, and Daniel Cohen-Or. Latent-nerf for shape-guided generation of 3d shapes and textures. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Oscar Michel, Roi Bar-On, Richard Liu, Sagie Benaim, and Rana Hanocka. Text2mesh: Text-driven neural stylization for meshes. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Siddharth Ahuja or ahujasid. blender-mep: Blender model context protocol integration. https:
//github. com/ whwjjssddbbs http///bbedeemeeeeeeeeee 2025. Accessed: YYYY-MM-DD.
Paszke. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint 1912v:1 2019.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. In International Conference Learning Representations (ICLR), 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021. URL 00020.
12

---

Published as a conference paper at ICLR 2026
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Tianhe Ren, Shilong Liu, Ailing Zeng, Jing Lin, Kunchang Li, He Cao, Jiayu Chen, Xinyu Huang, Yukang Chen, Feng Yan, Zhaoyang Zeng, Hao Zhang, Feng Li, Jie Yang. Hongyang Li, Qing Jiang, and Lei Zhang. Grounded sam: Assembling open-world models for diverse visual tasks, 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Ruoxi Shi, Hansheng Chen, Zhuoyang Zhang, Minghua Liu, Chao Xu, Xinyue Wei, Linghao Chen, Chong Zeng, and Hao Su. Zerol23+ + a single image to consistent multi-view diffusion base model. arXiv preprint arXiv:2310.15/ 2023.
Yichun Shi, Peng Wang, Jianglong Ye, Mai Long, Kejie Li, and Xiao Yang. Mvdream: Multi-view diffusion for 3d generation. In International Conference on Learning Representations (ICLR), 2024.
Jaidev Shriram, Alex Trevithick, Lingjie Liu, and Ravi Ramamoorthi. Realmdreamer: Text-driven 3d scene generation with impainting and depth diffusion. arXiv preprint arXiv:240..07199 2024.
Jingxiang Sun, Bo Zhang, Ruizhi Shao, Lizhen Wang, Wen Liu, Zhenda Xie, and Yebin Liu. Dream- craft3d: Hierarchical 3d generation with bootstrapped diffusion prior. In International Conference on Learning Representations (ICLR), 2024.
Yifan Sun, Menglin Zhang, Yi Jiang, Xinyu Wang, Kai Chen, Ping Luo, and Dahua Lin. Layoutvlm: Indoor scene layout generation with vision-language models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2025.
Dmitry Tochilkin, David Pankratz, Zexiang Liu, Zixuan Huang. Adam Letts, Yangguang Li, Ding Liang, Christian Laforte, Varun Jampani, and Yan-Pei Cao. Triposr: Fast 3d object reconstruction 2024.
from single image. arXiv preprint arXiv:2403.08
Alexander Vilesov, Pradyumna Chari, and Achuta Kadambi. Cg3d: Compositional generation for text-to-3d via gaussian splatting, arXiv preprint arXiv:2311.17909 2023.
Haochen Wang, Xiaodan Du Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich. Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In IEEE International Conference Computer Vision (ICCV), 2023.
Peng Wang and Yichun Shi. Imagedream: Image- progeppromt multi-view diffusion for 3d generation. arXiv preprini arXiv:2312.022000 2023.
Xinjie Wang, Liu Liu, Yu Cao, Ruiqi Wu, Wenkang Qin, Dehui Wang, Wei Sui, and Zhizhong Su. Embodiedgen: Towards a generative 3d world engine for embodied intelligence, 2025. URL
abs http:////xxiivx../.aaa/
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolific- dreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In Conference on Neural Information Processing Systems (NeurIPS), 2024.
Dejia Xu, Hanwen I Liang, Neel P Bhatt, Hezhen Hu, Hanxue Liang, Konstantinos N Plataniotis, and Zhangyang Wang. Comp4d: LIm-guided compositional 4d scene generation. arXiv preprint 1639:22403.1693, 2024a.
Yinghao Xu, Hao Tan, Fujun Luan, Sai Bi, Peng Wang, Jiahao Li, Zifan Shi, Kalyan Sunkavalli, Gordon Wetzstein, Zexiang Xu, and Kai Zhang. Dmv3d: Denoising multi-view diffusion using 3d large reconstruction model. In International Conference on Learning Representations (ICLR), 2024b.
Han Yan, Mingrui Zhang, Yang Li, Chao Ma, and Pan Ji. Phycage: Physically plausible composi- tional 3d asset generation from single image. arXiv preprint 1..i:22111.185444 2024.
13

---

Published as a conference paper at ICLR 2026
Kaixin Yao, Longwen Zhang, Xinhao Yan, Yan Zeng, Qixuan Zhang. Lan Xu, Wei Yang, Jiayuan Gu, and Jingyi Yu. Cast: Component-alignn aligned 3d scene reconstruction from an rgb image. arXiv preprint ar arXiv:2502.11880 2025.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, et al. Scaling autoregressive models for content- rich rext-to-image generation. International Conference on Machine Learning (ICML), 2022.
Tianyuan Zhang, Hong-Xing Xing Yu, Rundi Wu, Brandon Feng, Changxi Zheng, Noah Snavely, Jiajun Wu, and William T Freeman. Physdreamer: Physics-based interaction with 3d objects via video generation. European Conference on Computer Vision, pp. 388-406. Springer. 2024.
Haoyu Zhao, Hao Wang, Xingyue Zhao, Hongqiu Wang, Zhiyu Wu, Chengjiang Long, and Hua Zou. Automated 3d physical simulation of open-world scene with gaussian splatting. arXiv preprint
arXiv:2411.12789 2024.
Junwei Zhou, Xucting Li, Lu Qi, and Ming-Hsuan Yang. Layout-your-3d Controllable and precise 3d generation with 2d blueprint, 2025a. URL https: org/ abs 2410. 15391.
Xiaoyu Zhou, Xingjian Ran, Yajiao Xiong, Jinlin He, Zhiwei Lin, Yongtao Wang, Deqing Sun, and Ming-Hsuse Yang. Gala3d: Towards text-to-3d complex scene generation via layout-guided generative gaussian splatting. arXiv preprint arXiv:2402.07207, 2024.
Yang Zhou, Zongjin He, Qixuan Li, and Chao Wang. Layoutdreamer: Physics-guided layout for text-to-3d compositional scene generation. arXiv preprint arXiv:2502.01949, 2025b.
Zhizhuo Zhou and Shubham Tulsiani. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
14

---

Published as a conference paper at ICLR 2026
A
PSEUDO-CODE OF BUILDING SCENE TREE
Algorithm Build Scene Tree
Require:
Scene objects
Root node V (ground)
Ensure:
Hierarchical scene tree T:
1: Initialize tree T with root node 9;
2: Mark all objects in O unvisited;
3: Procedure BuildSceneTree(mm
4: for
where is unvisited do
5: if o is in contact with n and has an physical dependency relation with n then
6: Add as child of in T:
7: Mark as visited;
8:
Call BuildSceneTree(O))
9: end if
10:
end for
11: Call BuildScene
B DIFFERENTIABLE SIMULATION DETAILS
B. FORWARD SIMULATION
We model each object in the scene as stiff affine body Lan et al. (2022), where any point on the object with an initial position Xinit undergoes an affine transformation to its current position AXinit P. where A E 1x3x3 is transformation matrix and is translation vector. Together, they define the degrees of freedom (DOFs) of the object as q= [p. A] E 13x4
To simulate the motion and contact of the objects, we employ a custom GPU-optimized affine body dynamics (ABD) simulator based on Huang et al. (2025a). The simulator solves for the configuration gu+1 E 12N at time step + 1, formed by flattening and stacking the DOFs of all N objects, from the configuration at the previous time step via:
(4) Here, M the mass matrix, and T = In + A&g the predictive state used in artificial time stepping, which omits velocity. At denotes the simulation time step, and is the gravitational acceleration. The potential vi models stiff elasticity to preserve object shape, B is barrier potential enforcing non-penetration constraints, and D is semi-implicit friction potential following Li et al. (2020).
M qn)) + At2 (v++())+) + VB((n+1) + VD 9n)) 0.
B.2 BACKPROPAGATION
To optimize the initial layout go, compute the gradient of the loss function L w.r.t go using the chain rule:
dL d-=(a (Dal) dgo Ago
Oq2
dqn
0qn+1 Oqn
dL
(5)
0qu
Zqn-1
dqa+1
Here, can directly computed at the target step n automatically obtained via Py- dgn+1 Torch Paszke (2019). The key step lies in computing 0g. 11, which we derive using implicit differ- By.
entiation. Rewriting Equation yields:
4n+1 = In + At M
(f(n=1)) + Mgl.
(6)
where f(gu+1) V BB(qn+1) V D(gn41, qu). Differentiating both sides of Equation 6 with respect to and isolating the derivative yields:
0qn+1 0qn
8 DDD(qn+1.9)) APM
T
2
(7)
3qn+1
0qn+10g
15

---

Published as a conference paper at ICLR 2026
Substituting Equation 7 into the chain rule expression in Equation allows us to compute the full gradient dE
den which we use to update the initial layout. Both forward simulation and backpropaga- tion are fully GPU-accelerated for computational efficiency.
B.3 PHYSICAL AND ALGORITHMIC PARAMETERS
All simulations performed using differentiable rigid body simulator with the following physical and algorithmic parameters. They are generally applicable to rigid body scenes and only a few of them needs tuning. We define the mms, i.c. the mean mesh size, be the average of the longest side of the bounding box of all meshes.
Fixed parameters:
Friction coefficient: A Coulomb friction coefficient of 0.2 is used for all contact interac- tions.
Normal contact stiffness: The genalty-based normal-force model uses an effective stiff- ness of 1.0 GPa, corresponding to rigid-body behavior.
Gravity: Standard Earth gravity (0, 9988, 0) ms is used.
Time step (At): The simulation integrated using time step of 0.03
Newton solver tolerance: In each time step. Newton's method applied to solve the time integration problem, and the termination criteria is the velocity residual falls below
0.1 ms-
Maximum number of frames: This specifies the duration of the simulation. We use 300
frames for all the scenes.
Optimization learning rate: We learning rate of 0.001 for simulation-in-tth-lloo optimization.
Maximum optimization epochs: Differentiable simulation is allowed up to optimiza- tion iterations.
Tunable paramters:
Contact distance threshold: A threshold of m.0lmms is used for collision handling in most cases. Two objects are considered in contact when their distance falls below this value. We use value of 5 x 10-4 for the stackedblocks scene to capture the intricate balaneing behavior.
Friction velocity threshold: Relative velocities below 0.01 are modeled to generate static friction forces for most cases. We use a value of 10 for the stacked blocks scene.
Optimization frame interval: We compute semantic losses every 10 frames by defult and accumulate it during the simulation. This parameter are set according to the frequency of contact events in each simulation. Most of examples achieve perfect semantic align- ment at the first optimization iteration, and thus no tuning needed.
B.4 TIMING AND MEMORY CONSUMPTION
All experiments are conducted on single NVIDIA A5000 GPU. The average end-to-end time to generate scene is 1632 seconds. The main computational cost arises from object generation, which requires an average 762 seconds per scene. During simulation-in-the-lon stage, each optimization iteration takes approximately 30 seconds, and use 50 iterations in total, selecting the solution with the lowest objective value.
For all generated scenes, the simulation fits within 8 GB of GPU memory. In practice, memory usage scales with the size of the contact graph, which depends on the geometric complexity of the objects and the number of active contacts in the scene. Our formulation relies sparse matrix representations for both system dynamics and contact operators, which helps maintain a moderate memory footprint.
Given ongoing improvements in GPU hardware and simulation techniques, we do not anticipate efficiency and memory to be the limiting factors for typical applications.
16

---

Published as a conference paper at ICLR 2026
C IMPLEMENTATION DETAILS
We implement our proposed algorithm in Python on Ubuntu 20.04. In the layout optimization, we leverage Libupic Huang et al. (2025a) simulation platform. The optimization performed using ADAM Kingma (2014). Scenes that are already semantically consistent after the first simulation are not optimized. Our algorithm is deployed and run on a single NVIDIA RTX 4090 GPU. All our baselines were tested on the NVIDIA A6000 GPU.
D TEXT PROMPT USED IN ABLATION STUDY
The complete text prompt used in our ablation study are follows.
Figure 6: "On the left side, there is metallic cylindrical pen holder containing two black pens, a wooden ruler, and pair of gray-handled scissors. On the right side, there is a neatly stacked pile of three books with red covers and visible pages. The items are placed on a light wooden surface, and the background is plain white, creating a bright and simple composition."
Figure "a stack of colorful wooden blocks arranged vertically, featuring red, blue, yel- low, green, orange, and purple pieces, balanced on f surface.".
Figure 9 "A brown leather sofa decorated with plush toys, including two large teddy bears, a gray elephant, white rabbit, a yellow giraffe, and two throw pillows, sits in cozy room with two round burgundy floor cushions in front."
E
MORE EXAMPLES
A boby bunny sitbing top of seack of poncoles-
Al Welzard standing Wooden Ssil parched
Aeciont Books perched stop
the Wloden Dark
Text prompts
(a) GraphDreamer
(b) Blender-MCP
(c) MIDI
(d) Ours
Figure 10: Comparison of generated scenes from text prompts used in GraphDreamer Gao et al. (2024).
A cany carkoon setap with bed, bedside cable, lamp, booksheif, and small
OOM
A worm setup with a bed, bedside cables, plant, few books, and small lamp
Text prompts
(a) GraphDreamer
(b) Blender-MCP
(c) MIDI
(d) Ours
Figure 11: Comparison of generated scenes from text prompts used in MIDI Huang et al. (2024).
17

---

Published as a conference paper at ICLR 2026
In addition the 18 text prompts used for comparison with the baseline, further tested our algo- rithm on 12 additional examples, as shown in Figure 12. All of these prompts yielded semantically accurate and physically stable results.
F METIRCS
Clip score and VQAScore These two metrics measure the semantic similarity between the ren- dered scene images and the corresponding input text prompt. Specifically, we render the scene from 18 viewpoints by sampling three depression angles (09, 20", and 45%) and six evenly distributed horizontal angles. These rendered images are then used to compute the Clip score and VQAScore.
Simulated Scene Displacement (D) This metric computes the normalized average displacement of object vertices in the scene before and after applying a simulation. To ensure consistency across different baselines,
first normalize all scenes such that their bounding box diagonal length equals 2. We then compute the displacement of every vertex over all simulated frames and aggregate these values into a single scalar quantity. Finally, this scalar is normalized by the total number of vertices and by the scene's diagonal length. Formally, let V/ denote the position of vertex j at simulation frame i, and let V and T denote the number of vertices and frames, respectively. The metric is defined as the average per-vertex displacement normalized by the scene diagonal length l:
T
D
Σ
(1) 1i
1 o
Ratio of Penetrating Triangle Pairs (R) This metric quantifies the extent of penetration between objects in the scene, serving indicator of the scene's geometric correctness. We first normalize all scenes such that their bounding box diagonal length equals 2. We then remesh each object using fTetWild Hu et al. (2020), setting the target triangle edge length 0.05 times the scene diagonal. After normalization, we compute R as the ratio between an approximated total length of intersection
contours and the length of the scene diagonal I: TTe (T-E Tp.clfe, where Tp is the total number of penetrating triangle pairs in the scene, Tp.s is the number of self-penetrating triangle pairs in object i, which excluded, and le is the average edge length of all object meshes.
Physical Plausibility Score This VLMe-based metric evaluates the physical plausibility of gener- ated scene by asking a GPT model to score the realism of object contacts and physical relationships in the rendered image. Specifically, we use the following prompt:
"The semantic meaning of this scene is: '.... Please evaluate whether the physical relation- ships in this image reasonable and whether the contacts between objects physically realistic. Give this scene physical plausibility score from 0 to 100."
G MORE COMPARISON
We qualitatively compare our method with the baselines the text prompts previously presented in MIDI and GraphDreamer, as shown in Figure 11 and Figure 10, respectively. For the comparison with MIDI, since it requires prompts in their paper and use these as MIDI's inputs. For the other baselines, we directly use the text prompts. For our method, we use the same text prompts while ensuring that the reference image is consistent with the input image used for MIDI.
image input, we first generate images from the provided text
H COMPARISON WITH LLYOUU-YOUR-S
Using the same text prompt as in Layout-Your Your-3D Zhou et al. (2025a), we compare a generated scene to highlight our improved physical plausibility. In Figure 13(a), PAT3D produces physically accurate contact. In contrast, in Figure 13(b), taken from the Layout-Your-31 paper, the basketball intersects the sofa, resulting in geometry penetration that precludes intersection-free simulation.
18

---

Published as a conference paper at ICLR 2026
1 DISCLOSURE
We made use of LLMs to polish writing. We made sure that our input text to LLMs will not be used for training purposes.
19

---

Published as a conference paper at ICLR 2026
stlver taptop with its screen turned off is placed on a small round wooden stoot with three legs."
"A table with a basket eggs on
"A small Infiatable blue wndtt kiddle pool filted with eight colorful beach balls with red, yellow, and white pancls float on the surface.
"A round white plate holds a simple arrangement of food: two brown sausages placod side by side, soveral pleces bright yellow com on the cob cut into chunks, and thrse broczal florets."
baskets pears, apples, alongeid jam Jars."
cylindrical holder containing
seseral stat loneryy Items, incuding pair Haad-handledd scisoon white ruler pan, and mechanizal penall, neatly stacked plle of three hardcower books earthy tones orange, belge, blue."
lies
"A traroparent glass supports stack of two beige bowis and two beige plates, with sliver fork standing upright on the topmost plate."
"On the desk, black desk lamp sits the lift, accompanied by white mug filled with pencils. The right side dominated by tall stack hardcover books, with ummall putted plant adés touch of greenery right. A black leather office chatris centered behind deck."
"A small chocolate cake tapped with swid white whipped cream and decorated with two frwshh red strew berries, sersed mund whibe plate plaord on wooden table."
book top, while
"A buik basket containing one banana and two apple""
"An inflatable swimming pool with toy ducls, boot, and starfish toy."
"A toy car on four books."
Figure 12: More results of our method.
20

---

Published as a conference paper at ICLR 2026
PAT3D
Layout-your-3h
Figure 13: Comparison between PAT3D and Layout-your-3D. (Text prompt: "A brown sofa with cushion, a teddy bear, and a basketball on it")
21


Reviews:
[{"review_id": "W8H7jELACu", "reviewer": "Reviewer_D8i8", "summary": "PAT3D is a physics-augmented text-to-3D scene generation framework that overcomes common issues in geometry-only layouts (floating, unstable stacks, wrong supports) by integrating differentiable rigid-body contact simulation into the pipeline. From a text prompt, it synthesizes a reference image, generates segmented object meshes, and uses a VLM to infer inter-object physical dependencies, forming a hierarchical scene tree. It then constructs an intersection-free, physics-aware initialization by inserting small gravity-aligned gaps for support pairs, and lets objects settle via simulation before optimizing the layout with differentiable, artificially time-stepped dynamics for semantic and physical consistency. Experiments on contact-rich scenes show state-of-the-art visual quality, semantic alignment, and physical plausibility, with scenes remaining editable and directly usable for simulation-based robotics evaluation.", "questions": "Please check Weaknesses", "limitations": "", "rating": 6, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- More realistic physics: Differentiable rigid-body contact simulation reduces floating, interpenetration, and unstable stacks.\n- Better semantic–layout alignment: A VLM infers object dependencies to build a scene tree that preserves text-described spatial relations.\n- Stable, collision-free initialization: Physics-aware initialization with small gravity-aligned gaps improves convergence and avoids intersections.\n- Optimizable and editable scenes: Layout is refined via differentiable simulation; outputs remain easy to edit and interact with.", "weaknesses": "- Scalability to very large scenes: Many-object, dense-contact scenes increase solver complexity (contact graph size), potentially impacting stability and runtime.\n- Lack of quantitative study in ablation:   Measurable gains introduced by each component are necessary, making audience further understand the contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zO38Eb4acd", "reviewer": "Reviewer_etTa", "summary": "The paper introduces PAT3D, a framework for generating 3D scenes from textual prompts, with emphasis on physical plausibility and simulation-readiness. \nThis work uses a 3D Objects and Spatial Relation Extraction module by generating reference images, a Layout Initialization module with the help of point clouds, scenetree, and a Layout Optimization module with self-defined physical constraints to generate simulatable and physically plausible 3D scenes.\nIn short: PAT3D bridges text→3D scene generation with physics simulation, aiming not just for plausible appearance but also for physically stable, actionable 3D scenes.", "questions": "- What is the success rate of this work? Will the simulation finally converge to a plausible point with multiple iterations? Or it is largely dependent on the initialization?\n- Are there more failure cases? This failure case is not that typical, and I guess it can not reflect the major limitation of this work.\n- How much time does it take to generate a single 3D scene with the simulation-in-the-loop? What is your edge over [1]?\n\n[1]: Embodiedgen: Towards a generative 3d world engine for embodied intelligence", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The visualization results are obviously better than previous works.\n- The pipeline is intuitive and direct. With the help of depth and point cloud, this work is able to obtain precise 3D scene layouts and spatial relationships.\n- Applying simulations in the process of 3D scene generation is an interesting method, which alleviates a lot of trouble in the nuisance 3D layout optimization process.", "weaknesses": "- The novelty of this work is relatively weak. The '3D OBJECT AND SPATIAL RELATION EXTRACTION' part also appears in [1], and the novel parts are the simulation instead of 3D scene optimization, which is a little bit weak.\n- The use of rigid-body simulation and “simulation in the loop” likely requires non-trivial compute and careful tuning. The reproducibility, speed, and resource requirements might limit broader adoption.\n- Too few quantitative results, since the paper features the simulation ability of this work, I guess some more metrics that can reflect the physical plausibility and the advantage for manipulations are needed.\n\n[1]: Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LJ80wi7BbY", "reviewer": "Reviewer_kPCZ", "summary": "This paper introduces PAT3D, a physics-augmented framework for text-to-3D scene generation that addresses the critical gap of physical plausibility in multi-object scene synthesis. The technical approach integrates vision-language models with differentiable rigid body simulation to produce scenes that are both semantically consistent and physically stable. The method decomposes the generation pipeline into three stages: object and spatial relation extraction via VLMs, physics-aware layout initialization using hierarchical scene trees, and simulation-in-the-loop optimization. Experiments demonstrate improvements over existing methods in physical stability metrics and semantic consistency.", "questions": "Generally I think this is an important problem and the results look interesting. But even in the two examples (which are very limited), the simulation looks a bit unrealistic. So I'm not sure if the technical approach is really addressing the challenges stated in the paper.\n\nAlso, I'm not fully convinced why the work is different from CAST.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- This paper focuses on a meaningful and underexplored problem. The lack of physical plausibility in text-to-3D scene generation is important for downstream applications in robotics and simulation.\n- Demonstrates practical applications in scene editing and robotic manipulation tasks.\n- Clear experimental validation showing zero interpenetration and stable equilibrium states.", "weaknesses": "- Limited video demonstrations make it difficult to fully assess the dynamic behavior and stability of generated scenes under various conditions. The supplementary materials would benefit from more extensive video results. Currently there are only 2 examples.\n- Although the paper claims the first work on physics-plausible scene generation, I'm not sure how different the problem setting is from CAST. From the demo examples of CAST, they also have physics simulation to make sure objects are placed in a stable and non-penetrating way. How is the proposed work different from it?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)