HoloScene: Simulation‑Ready Interactive 3D Worlds from a Single Video
Authors: Hongchi Xia
Chih-Hao Lin
Hao-Yu Hsu
Quentin Leboutet
Katelyn Gao
Michael Paulitsch
Benjamin Ummenhofer
Shenlong Wang
Venue: NeurIPS 2025 poster ()
Keywords: computer vision
digital twin
simulation
Primary area: deep_learning

Abstract: Digitizing the physical world into accurate simulation‑ready virtual environments offers significant opportunities in a variety of fields such as augmented and virtual reality, gaming, and robotics. However, current 3D reconstruction and scene-understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealistic rendering, or realistic physical properties for reliable dynamic simulation. To address these limitations, we introduce HoloScene, a novel interactive 3D reconstruction framework that simultaneously achieves these requirements. HoloScene leverages a comprehensive interactive scene-graph representation, encoding object geometry, appearance, and physical properties alongside hierarchical and inter-object relationships. Reconstruction is formulated as an energy-based optimization problem, integrating observational data, physical constraints, and generative priors into a unified, coherent objective. Optimization is efficiently performed via a hybrid approach combining sampling-based exploration with gradient-based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness.

HoloScene: Simulation-Ready Interactive 3D Worlds from a Single Video
Hongchi Xia'
Chih-Hao Lin'
Hao-Yu Hsu'
Quentin Leboutet
Katelyn Gao²
Michael Paulitsch
Benjamin Ummenhofer Shenlong Wang!
University of Illinois Urbana-Champaign
Intel
Input Video
Simulation Ready and Interactive digital twin from single video
Resl Time interoctive Game
Oripler
Editing
Inmergae Expli ence Recowding
Platial Plaalie
Scene Graph
Dyuai Voas Blkcsb
Table
Bachground
Figure Overview of HoloScene: From single input lideo-along with visual such as segmentation and monocular depth--HoloScene reconstructs a simulation-reads. interactive 3D digital twin represented scene graph with complete geometry, physically plausible dynamics, and realistic rendering. The resulting model enables variety of downstream applications, including real-time interactive gaming, 3D editing, immersive experience capture, and dynamic visual effects.
Abstract
Digitizing the physical world into accurate simulation-ready virtual environments offers significant opportunities in variety of fields such augmented and vir- tual reality, gaming, and robotics. However, current 3D reconstruction and scene- understanding methods commonly fall short in one or more critical aspects, such as geometry completeness, object interactivity, physical plausibility, photorealis- tie rendering, or realistic physical properties reliable dynamic simulation. To address these limitations, we introduce HoloScene, novel interactive 3D recon- struction framework that simultaneously achieves these requirements. HoloScene leverages comprehensive interactive scene-graph representation. encoding object geometry, appearance, and physical properties alongside hierarchical and inter- object relationships. Reconstruction is formulated problem, integrating observational data, physical constraints, and generative priors into unified, coherent objective. Optimization is efficiently performed via hybrid approach combining sampling- baseed exploration with gradient-bas based refinement. The resulting digital twins exhibit complete and precise geometry, physical stability, and realistic rendering from novel viewpoints. Evaluations conducted on multiple benchmark datasets demonstrate superior performance, while practical use-cases in interactive gaming and real-time digital-twin manipulation illustrate HoloScene's broad applicability and effectiveness. Project page: here
energy- optimization
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Imagine wanting, decades later, to revisit the home you live in and love today-hoo how would you capture its memory? Photographs and videos record authentic details but lack immersion; 3D Gaussian splats or photogrammetry can be immersive, yet static chairs and tables feel lifeless. Ideally, we would digitize our environment into a fully interactive digital twin: complete, composable, photorealistic, and manipulable just like the real world. Our work takes a step toward this goal by enabling users to create in silico twins of their surroundings from single video.
Digitizing the physical world into a simulation-ready virtual environment offers immense oppor- tunities in augmented and virtual reality, gaming, and robotics. However, despite advances in 31) modeling and scene understanding, key challenges remain: capturing complete geometry and ap- pearance in occluded regions, inferring inter-object relationships, and ensuring physical plausibility
and interactivity. Existing Real2Sim methods produce incomplete geometry [88] 45
71 or unstable
physics 777 75 existing amodal reconstruction focuses single-image setting [86] I individual
objects 76
15 39)
neglects physical plausibility [49] or relies on asset retrieval
10 sacrificing
fidelity and practicality; and prior physically plausible reconstruction
48
15
is limited simple
object-scene interactions or requires full observations.
To address these gaps, we introduce HoloScene, an interactive 3D reconstruction framework that unifies geometry completeness, object completeness, physical plausibility, realistic rendering, and physical interaction. HoloScene represents scene as an interactive scene graph encoding object geometry, appearance, and physical properties in a hierarchical structure. We cast scene -raee recov- ery from video as a structured energy-base optimization, integrating observational data, physical constraints, and generative priors into a single objective. To solve this challenging problem, we propose novel divide-and-conquet strategy combining sampling- sraplinng-asddd tree-structured search with gradient-based refinement. The resulting scene models exhibit complete, accurate geometry; stable physical interactions; and realistic rendering from novel viewpoints.
Experiments on three challenging benchmarks demonstrate superior geometry accuracy and physical plausibility, with rendering performance comparable to state-of-the-ar amodal and physics-awar reconstruction methods. We further showcase HoloScene's 's versatility through practical applications in interactive gaming, realistic video effects, and real-time digital-twi -tiin manipulation.
2 Related Works
Interactive 3D Scene Model
Visnal
Real-ime Rendering
Twin Fidelity
Physies Capacity
Recent advances in 3D scene modeling [52 93
Methed
Inpur
3D Recon
Oprimization
17
5
68
re-
ACDD [0]
image
X
X
construct 3D scene from input images videos, representing
GorlDSR
imnge
PhysCloup 13
imnge
X
/x
Single Object Seme
Dilienatiable
CAST
image
/x
Differentiable
the scene as neural fields [46, 65] 61
vidoo
BakedSDFF
X
Video
X
19 62
25
3 signed dis-
ObjerSDF++
tance functions (SDF) [7] 87
Video2Came
[
video
Single Object
DRAWER
vidoo
Single Object Objects
51 53
88
91
29 and 3D Gaus- 92 90). While pro-
PhyRecon
sians 23
20
DR-Rtoon 40
video
HoloScene
video
Scone
Dift Sampling
ducing realistic renderings from novel views, these works cannot provide 3D assets that allow interactions (e.g, move the chairs to different poses). Reconstructing realistic and interactive en- vironments from real images and videos remains challenging due to limited observation, occlu- sion, and physical reasoning. Some previous works reconstruct 3D objects from sparse view- points 39 35 79 7 4T, and some estimate physical properties from visual observation 9 [5). Nevertheless, these works focus on object-level tasks and cannot handle large and complex indoor scenes. PhyRecon (48] optimizes stable 3D scenes with differentiable physical engines, but does not model inter-object interactions and realistic appearance. DP-Recon 49 Video2Game [77].
Table 1: Comparison of Interactive 3D Scene Models.
Drawer 78 leverage generative prior [[] (55) and foundation models (59
60 to reconstruct de-
composed 3D scenes. However, these works can only produce limited components or lack physical stability. Text-to-3D generation [
33 72 creates diverse layouts from text prompts, but fails
to reconstruct digital twins that faithful to visual inputs. In this work, propose reason

---

object interaction with the scene graph, and utilize generative priors and a novel sampling strategy to reconstruct the geometry and appearance of every component, constructing realistic, physically plausible, and interactable 3D environments.
Data-driven Simulation Simulation plays a pivotal role across robotics, self-driving, and content creation, but building high-fidelity virtual scenes remains costly, and the sim-to-real gap poses great
challenges. To address this, data-driven simulation 2 g
63 84] has emerged, enabling the
modeling of physical dynamics 38 28
80
21
13
22
95 lighting conditions [3 [88
31], and action-
conditioned outcomes [21] 36 6
77 22,
37
78 42 directly from real-world data. These methods
have also been applied in robot learning 8 44 57
35 84 LiDAR simulation 34 4
84 99
98). and interactive media [88 77 In robotics, related real-to-sim approaches (9
10
67
70
27
reconstruct interactable environments from the real world for reproducible embodiment. However,
they still lack physical realism. Recent works 48 6 [97] leverage differentiable physics priors in reconstruction, but they neglect complex inter-object relationships The closest work to ours is CAST (86). which also considers physical plausibility in scene reconstruction. However, our proposed HoloScene differs fundamentally in three key aspects: (1) Problem formulation: CAST takes a single-ing -mage generative approach where fidelity input appearance and geometry is not enforced, while HoloScene performs multi-view joint optimization from videos, explicitly ensuring faithful reconstruction of observed scenes through inverse neural rendering.
(2) Physical stability: CAST's differentiable optimization only avoids penetrations but cannot guarantee long-tern stability. HoloScene employs kimulator-in-the-bee physically stable over time. (3) Inference strategy: CAST follows multi-step pipeline with greedy sequential optimization, whereas HoloScene adopts unified energy- formulation (Eq. () that jointly optimizes observation fidelity, physical plausibility, and object completion through generative sampling and scoring. We compare HoloScene with prior works in Tab. I
optimization with Isaac Sim [47 ensure objects remain
3 Method
Given the observations O == (O which include the input video sequence {Ie), camera poses { (inferred or ground truth), and instance masks {M&} (inferred or ground truth), our goal is to reconstruct realistic, complete, and physically plausible digital twin of the input scene, yielding interactive, sim-ready assets compatible with simulators and game engines, and which can be used to generate novel visual content. To this end, we represent the scene graph representation that encodes object geometry, appearance, physical properties, and hierarchical inter-object relationships (Sec. (.I)) We combine observational evidence, generative priors for shape completion, and physical simulation for stability to formulate scene-graph recovery as an energy min- imization problem (Sec. B.2p. Finally, we propose an inference method that integrates sampling- tree search with differentiable optimization (Sec. 3.3). Fig summarizes our approach.
interactive 3D scene
3.1 Scene Representation
We represent the scene an interactive 3D scene graph 9 = (V, E). Each node v, E [v,) 2 represents either the background scene or one of the N objects present. A node v, = (gi, f. P;. T,) is comprised of geometry g:, appearance f,, physical properties p,, and dynamic states T, Each edge ei.j = (Vi, V2) E encodes an object-object relationship in G.
Geometry: We represent the geometry of each node v; in the scene with an instance-level neural SDF (x; 0): 3 + R. where x E g is any point in space and 0 are Iearnable parameters. Additionally, maintain mesh representation M,
to facilitate physical simulation and efficient rendering, MarchingCube(g.) for each object, extracted from its SDF using the marching cubes algorithm.
For each object Vi. we encode appearance f; = (c,, a; M4, Σ₂) as Gaussian splats, enabling real-time, high-quality rendering. C;, Qg M4, Σ, are color, opacity, mean, and covariance of Gaussians, respectively. Gaussians capture finer detail than colored meshes but hinder consistency between appearance, geometry, and simulation; following recent work 78), we adopt a Gaussians- on-Mesh (GoM) approach and attach each splat to its mesh to ensure alignment and enable physical interactions. Given camera intrinsics K and extrinsics &, we denote the splat-rendered RGB images, masks, depth and normal maps I, M, D, N SplatRender(G; K, E)
Appearance:
3

---

Stage 1: Gradient-BBal Baseet Optimization for Init
Stage 2: Generative Sampling & Tree Search for Geometry Physical Plausibility
Stage 3: Final
Finetune Realism
Pilkow Sample +1
Render
Nllow
Safe
Sumpll
Eors
Gradient
Room
Eypunle
Background
Physical Stable Scene via Recursive Tree Sampling
Final Render
Figure 2: Overview of HoloScene Optimization Stages:
Given multiple posed images as well as some visual cues (instance masks, monocular geometry priors), we first employ gradient-based -bsee optimization as initialization. Then we adopt a generative sampling and tree search strategy along the topology of the scene graph obtain the complete geometry with physical plausibility. Finally, the final fine-tuning over the scene further enhances the realism of the reconstructed scene.
graph is modeled rigid body. Its physical parameters P: (mi, Kis r;) comprise mass m, friction (resistance to sliding against other surfaces), damping (energy dissipation during motion), and restitution r; (elasticity upon impact). These parameters are used in downstream physical simulations to model the object's response to external forces and its interactions with other objects and the background scene.
Physies: Each object in
Object States: All object intrinsic attributes above are defined in object centric coordinates and remain invariant under motion. To handle dynamic changes, encode each object's 's rigid body state by a rigid transform T; from its object-centric frame to the world frame. During static reconstruction, T, is fixed; in dynamic simulation, may vary time. Let T (T, ) denote the set of all object states, Gr the scene graph under those states, and G the scene graph under the static state during reconstruction.
Object Relationships: Each edge e links two nodes with one of three relationships: 1) support, where v; rests in stable equilibrium its unique parent ppati) under gravity (each object has exactly one such parent, so support edges form a tree in the static scene graph); 2) beside, where siblings (pa()) pi()) have touching surfaces, causing occlusions without hierarchy instability; and 3) collide, where contacts with nonzero momentum yield dynamic effects effecct--ignoce iggnreed during static reconstruction but employed in simulation. Note that the object relationship might change depending on its dynamic status during simulation.
Our 3D scene graph's 's distinguishing feature is its support for physical interactions. Formally, at time given the dynamic scene graph GT with current object states T as well as input action a', the next states are computed as
Interaction & Simulation:
T Sim(T', a, GT:),
where Sim is rigid-body physical simulator using the mesh (M:) as collision geometry. Here, a' can represent external inputs- forces, torques, control actions-applic to the objects at time

---

3.2 Problem Formulation
Our framework takes input observations O of static scene and recovers the scene graph 9 EV.
The resulting scene graph must (i) explain the observations well; (ii) be geometrically complete and plausible; and (iii) reflect the scene's 's static, physically stable nature. To this end, we cast the problem as structured energy- seergy-minimization problem:
min EI, + Emask (M,9) + Funono (D,g) Ecomp(() + Eegu + Ephysics (g) 9
(2)
observation terms
rogularization temms
For simplicity, we omit the hyperparameter linear weights for each term. Next, we discuss each energy term.
Observation Terms:
The observation terms quantify the discrepancy between the reconstructed
3D scene and the input observations. Let I, M,, D,, N, = SplatRender(( ; K, &,) denote the rendered RGB image, instance mask, depth map, and normal map at camera pose E.- We then define three energy terms: the RGB energy (Ig = Σ, LMSE I,) + LLPIP (I, It). where I, is the ground-truth color image and the loss combines MSE and L.PIS losses [96]; the mask energy
Ee CE (M,; M.), where CE is cross-entropy and M, is either given labeled mask 66 or one inferred via segmentation tracking (24); and the monocular geometry energy
Emack M, g)
Emono D 9 = >. I|,, - 2 2 + Luorm Dz). where N, and D, are monocular normal and depth priors and Lnorm is the scale- and shift-invariant L2 loss 9T.
Regularization Terms: Because videos only partially observe 3D scene, optimizing observations alone cannot yield complete, plausible, and physically valid reconstruction; we therefore impose generative, geometric, and physical priors as regularizers to enable fully interactive 3D scenes.
The completeness energy Ecomp encourages complete reconstruction of each object's 's shape despite the partial observations. Inspired by generative image-to-3D methods (39], for each object we syn- thesize virtual observations O = {I., D., N by "shooting" it from multiple virtual viewpoints with pretrained multi-view diffusion model Wonder3D [39]. Unlike the single object setting for most image-to-3D works, because our complex scenes often feature inter-object occlusions (e.g., a sofa covered by blanket), we first inpaint occluded regions using LaMa (66] before generating these views. Given the synthesized observations, we define the completeness energy as
Ecomp
((Eaask (īz, (v.)) (Mg (v.)) + Emono (v:))),
where Ergh: Emono: Emaek are the observation losses defined similarly in our observation terms, although they are measured at virtual viewpoints here.
The geometry energy Egro ensures geometry compatibility between each object, such that their geometry does not intersect with each other:
Egco (9) Σ (Epene ((::: + Epenc_ mosh (g::
(4)
The SDF-penetration term Epene_sdr sdf = ExER, Exsr (0, - -xx(x) - (x)) ensures no two object SDFs overlap, where R((i) (x E R arg min: (x(x) = i} is the set of points belong to instance i. Intuitively, if x lies instance then for any other instance k; gx(x) 2 -9: (x) must hold to prevent intersections. Similarly, each object's mesh should not intersect with any other object mesh. This can measured by measuring whether intersecting two meshes resulting empty set or not: Epene_mest mesh = 1(inter(M,, M,, + 0).
Finally, it is important to ensure that our recovered digital twin of the scene simulatable; hence, physical plausibility is crucial. To this end, we introduce physics energy, which measures physical plausibility via two terms:
Ephysics = Estable + Etouch = Diff (T, Sim(T, agravity 9) +
Mi) (5)
(K,))E
The stable term Estante() Diff(T, Sim(T, agravity 9)) quantifies translational and rotational
deviations of each object, with Diff (T, T') Σ, (Itrans (T, 'T)) + rra((T," () and Sim is
5

---

ObiSDF++
Reference Image
Phy Rocon
N/A
Geometry
DP-Reeoa
Ours
DF Roeon
Appearance
Ours
Object
Object
Object
Figure 3: Qualitative Comparisons on Object Geometry and Appearance Reconstruction: Our method delivers superior reconstructions by smoothly inpainting occluded regions with LaMa and completing invisible back-facing geometry with Wonder3D. Unlike baselines, our approach eliminates object interpenetration, ensuring physical stability during simulation.
the forward physical simulator step as defined Eq I low Estable indicates static equilibrium under gravity, i.e. scene remains static in the simulator. The touch term Etouch E(.)) dist(M;;
encourages each supporting pair (1 to make contact, dist is the Chamfer distance between meshes.
3.3 Inference
Optimizing the scene graph from Sec. 3.2 is challenging because mixes discrete variables (graph topology, object-object relations) with continuous ones (neural SDFs, Gaussians, and physical pa- rameters) and includes non-differentiable terms like physical stability. We therefore use a three-stage divide-and-congee -nd-uuuuee approach: first, recover initial geometry and appearance by minimizing the observation terms; next, refine shapes and physical parameters by minimizing the geometry and physics terms through generative sampling combined with structured tree search; and finally, fine-tune appearance by re-minimizing observation terms. This yields a fully plausible, interactive 3D scene (Fig. 2
Stage Gradient-based Optimization: We first optimize each object node's appearance a, and geometry to match the observations O via gradient-based optimization. Specifically, we minimize observation terms plus SDFepenntrttion regularization through differentiable volume render- ing-similar to neural SDF methods 75
to obtain per-instance SDFs 9i. Additionally, we recover small objects by balancing training samples across all instances. We then extract initial meshes M; via marching cubes and refine each object's Gaussians f, via splat rendering and RGB rendering, mask, and monocular geometry losses, yielding our dual scene representation per each instance [16].
74 30
Stage Sampling-baseed Optimization: The Stage scene model supports freeview rendering and accurate visible-region geometry but remains incomplete, non -on-sccical,, and non- non-interactive. Directly minimizing Erompletes Ephysies: and Egens however, challenging due to complex high- order interactions (e.g., multi-object physical interaction), intrinsic multi-modality (invisible regions admit multiple solutions), and non-differentiable components (e.g., mesh intersections, physics simulations). To address this, adopt approach that combines the diverse proposal capability of

---

generative sampling with the combinatorial optimization strength of tree search to minimize our structured objective.
Scene graph edges creation: After the gradient-based optimization, our framework infers the topology of the scene graph 9 from the instance meshes extracted from neural SDF, where edges & encode support relations in a tree rooted at vo (the background, c.g. the room). We build this tree by analyzing physical contact and support relationships between objects: each object is assigned parent based on which surface provides its primary support, as determined by the geometry and orientation of contact surfaces. We begin by identifying objects resting directly on the background, then recursively process the remaining objects in an order that respects the physical dependency structure -objettreobjects that support others are registered before those they support. Starting from Vo, we repeat until all observed instances have been added to the tree.
Generative sampling: We begin by sampling diverse, complete shapes for each instance: prompt Wonder3D's s multi-diffusion model with real-world observations and generate virtual views I; from various viewpoints. Thanks to its generative nature, resampling multiple times with different seeds yields diverse virtual observations even from the same viewpoints. Then, for each virtual observation, we minimize Ecomp independently, producing diverse set of 3D shape candidates per object.
Structured tree search: We have generated multiple complete shape samples per instance that all fit the observations, but it remains unclear which combination is most physically plausible. Exhaustively evaluating every combination is impractical, since the thysical-plausibee high-order combinatorial optimization. To address this, we perform a tree search over our generative samples. Starting at the root node, traverse each node in breadth-first order; at each active node (object), we evaluate Ephysics for all samples and retain the sample with the lowest energy among those evaluated. We then adjust its state and physical parameters to enforce stability and prevent interpenetration (see details in the supplementary material).
energy Ephysics entails
Remark: The key novelty and advantage of the proposed inference algorithm is the combination of generative sampling with a structured tree search for amodal. physically plausible reconstruction. Unlike scene-level amodal methods [10] that rely asset retrieval, our sampling is asset-free and generates input consistent, diverse shape hypotheses. Unlike prior simulation verification
iimulation-veiification methods [49]
48], which only enforces object-ground consistency, our tree search ensures global stability along every support chain. By driving a con-differentiable simulator (e.g.. IsaacSim) end-to-end, we eliminate any reconstruction-to-dde do-plooommee gap.
Stage 3: Gradient-based Refinement Since Stage adjusts object states, physical parameters, and shape, it is necessary to further refine the Gaussians attached to the surface to ensure complete and realistic appearance. To this end, we fine-tune Gaussians for all objects using splat rendering by minimizing the observation terms via gradient descent. This yields our final scene graph.
4 Experiments
Dataset:
We conduct the experiments across multiple datasets: scenes from Replica (64. scenes from Scannet++ [89]. 2 scenes from iGibson (26), and one self-captured scene. The Replica and Scannet++ datasets cover diverse indoor structures and lighting conditions, and the iGibson dataset offers complete geometry of every object, allowing per-object reconstruction evaluation. Instance masks are provided by dataset annotations or estimated with SAM (4
Metrics: We evaluate geometry quality with Chamfer Distance (CD), F-Score (F1), and Normal Consistency (NC) (91], and assess rendering quality using PSNR, SSIM, and LPIPS. For physical evaluation, we adopt consistent physical parameters, put all scene components the Isaac Sim [47. and measure stability with translation/rotation changes when gravity is applied. The stability ratio is calculated as: Stable %
#Stable Instunces
where instances are identified stable if changes are under
# All Instances
certain threshold. We also report the object reconstruction ratio: OR% # Reconstrreteed Instances which quantifies how many objects are present in the final reconstruction, regardless of their completeness.
#All Instances
Baselines:
We evaluate our framework against SOTA approaches in instance-aware amodal 3D scene reconstruction. ObjectSDF++ 75l uses per-instance SDF for scene representation. PhyRecon 48 extends instance-aware scene reconstruction by incorporating differenniable physical loss to
7

---

no physies
Geoonetry
Reference Image
with physiccs
Objectsdi++
PhyRecon
Recon
Appearance
DP-Recom tho physics)
DP Recon (with physics)
Okis (no physics)
Ours (with physics)
We compare geometry layouts and appearance before and after physical simulation, with the table geometry reconstructions highlighted in inset figures. HoloScene's 's complete, non-interpenetrating geometry remains stable in physics simulators, unlike baseline methods. Our Gaussian mesh delivers high-quality, real-time rendering throughout the simulation process.
Figure 4: Qualitative Comparisons on Physical Simulation:
Geometry
Rendering
Physies
Method
Real Time
Stable (Ground) %
Stable (All) %
CD1
FIf
NC1
PSNR1
SSIM
LPIPSL
OR%
Replics
ObjSDF++
6.72
64.36
88.53
29.12
0.851
0.355
X
98.6
78.3
39.4
PhyRecon
71.07
0.434
DP Recon
3.45
87.66
94,23
22.10
0.728
0.420
56.3
21.7
8.5
Ours
4.05
83.21
92.21
27.82
0.849
0.304
100.0
95.7
81.7
Sccne
Scands++
ObjSDF++
25.20
70.71
87.15
27.46
0.887
0.292
X
96.5
81.6
28.2
Phy Recon
31.16
39.57
82.28
22.32
0.791
0.432
92.9
67.3
9.4
DP-Recon
22.96
65.48
87.13
21.44
0.715
0.466
90.6
20.0
9.4
0.268
ObjSDF++
38.64
83.74
0.299
65.0
iCibson
PhyRecon
11.27
45.49
83.85
27.40
0.860
0.333
62.9
45.3
5.2
DP-Recon
30.31
21.89
70.81
21.94
0.728
0,432
74.2
16.3
12.00
34.15
82.91
25.88
0.854
0.301
100.0
74.4
71.1
ObjSDF++
3.52
79.03
75.30
11.03
0.571
0.134
65.0
44.2
36.1
Object
iGibsan
PhyRocon
5.47
70.71
71.89
8.92
0.609
0.250
62.9
45.3
5.2
DP-Recon
61.31
70.61
13.90
0.770
0.301
Ours
3.17
81:31
78.13
16.55
0.863
0.185
100.0
74.4
71.1
Table 2: Quantitative Results Scene Reconstruction: HoloScene's generative sampling and scene graph-base based tree search produce the most physically plausible reconstructions while preserving high-quality geometry and supporting real-time and realistic rendering. We highlight the best and second-bee methods with distinct colors. For rendering quality, we only compare with DP-Recon's 's texture mesh-basee based rendering since ObjSDF++ and PhyRecon lack real-time rendering capabilities.
optimize unstable objects. However, it only handles object- ground interactions and does not support hierarchical or inter-object relationships. DP-Recon [49] incorporates diffusion Score Distillation Sampling (SDS) [55] for amodal sparse-view reconstruction. However, it does not handle inter-object occlusions. All three methods
instance-level SDFs, the SDF values of different instances might interfere with one another. We adopt their open-source codes and adapt them for the testing benchmarks. Please refer to the supplementary material for more implementation details.
Implementation details: Our inference pipeline consists of three stages. Stage employs gradient- based optimization for 100k steps with loss weights Argb 1.0, Amask 0.5, Adepth 0.5. and Anormal = 0.1. Stage 2 uses sampling based optimization with Årgb = 2.0, Amask = 0.5. Adepth = 10.0, Anormat = 10.0, Apene 5.0, generating three samples per instance. Stage 3 refines Gaussians via gradient-based optimization with dr1 = 0.95 and Xxsim = 0.05. The optimization takes approximately hours, 4 hours, and 20 minutes stages 1, 2, and 3, respectively, on a single A6000 GPU. We utilize Marigold [4] for monocular depth and normal estimation.
8

---

Level
Dataset
TexMesh
GGM
Physical Energy
Sceme Graph
CD]
F11
NC1
PSNRT
SSIM
LPIPS).
OR% 1
(AlI)
Stable
TexMesh
x
x
3.50
88.00
92.30
26.45
0.810
0.329
100.0
43.7
Scene
Replica
GoM
GoM
/
x
x
x
4.32
3.50
80.19
88.00
92.10
92.30
27.14
28.30
0.839
0.845
0.334
0.349
100.0
100.0
69.0
43.7
GoM
4.05
83.21
92.21
27.82
0.849
0. 0.304
100.0
81.7
Object
iGibson
GoM
x
3
4.20
79.22
76.96
13.74
0.735
0. 0.204
100.0
41.2
GoM
3.17
81.31
78.13
16:55
0.863
0.185
100.0
71.1
Table 3: Ablation Study Model Design: We observe improved physical stability and object-level reconstruction with the help of generative priors, physical energy, and scene graph representation, and there is trade-off between scene-level reconstruction and instance-level physical stability.
4.1 Experimental Results
Scene-level evaluations: We first evaluate the reconstruction at the scene level. In Tab. 2 we compare our framework with the three baselines across Replica, Seannet+++. and iGibson dataset, with qualitative results shown in Fig. and Fig. E Our method achieves the best physical reconstruction re- sults in terms of both object reconstruction ratio and stable object ratio, while maintaining comparable scene-level reconstruction quality in geometry and appearance. Unlike baselines [75] 48) [5]] where small objects often disappear due to SDF interference from larger adjacent objects, our framework benefits from balancing training samples across all instances during the optimization stage I, recover- ing all instances in the scenes. PhyRecon assumes that all objects rest directly on the ground, which damages geometry when objects are supported by other objects. DP-Recon prioritizes completeness via SDS over physical stability, failing to adequately address object interpenetration. In contrast, our sampling-based optimization and completion approach yields the most physically stable results.
Object-level evaluations: We evaluate object-level reconstruction using the iGibson dataset, which provides complete ground truth geometry and appearance for each object. This evaluation especially challenging as it tests the reconstruction of occluded regions that models never observe. Our evaluation compares reconstructed geometry directly with ground truth and renders 6 viewpoints around each object to assess appearance quality. Results in Tab. [ demonstrate that method outperforms all basclines in reconstructing invisible and occluded regions, validating our framework's effectiveness in completing objects beyond directly observed surfaces.
Ablation study: Our ablation study (Tab. 3) reveals the contribution of each component. Switching from textured mesh to Gaussian rendering improves visual quality, while adding physics energy with generative priors enhances physical stability. The scene graph inter-object relationships further improve physics performance by better reconstructing occluded regions, especially those from support relationships. However, we identify trade-off between scene-level reconstruction accuracy and physical stability, where optimizing for physical plausibility may occasionally compromise pixel-alignment with original observations.
Failure rate analysis: We conduct a statistical analysis failure modes in geometry and physics reconstruction. In Tab. we compare HoloScene against reconstruction-basse baselines 775 across geometry failures on iGibson and physics failures on all three datasets, with failure criteria defined by F-score thresholds (70.0) and simulation stability metrics. Geometry failures include invalid surfaces, partial and oversized shapes, while physics failures arise from invalid shapes incompatible with simulators, overlooked object-object contacts, inter-object penetration causing repelling forces, and inaccurate contact modeling leading to positional drift. Our method achieves the lowest failure rates in both categories. Unlike ObjSDF++ (75) and PhyRecon (48) which lack shape priors and produce invalid geometry, our generative priors eliminate all invalid geometry failures through reliable structural guidance and balanced sampling. Re-Recon [50] emphasizes visual completeness over physical constraints, resulting in high penetration failures. Additionally, PhyRecon's ground-contact assumption overlooks contact pairs that can destabilize the simulation. In contrast, we integrate generative priors with physics-aware constraints directly into the reconstruction process, which substantially reduces both invalid shapes and inter-object penetrations, demonstrating the effecttveness of physical consistency in the optimization.
48 [50]

---

Metbod
Gounetyy Fullure (Gibsoo)
Physics Fallure
Becakdown Dussees)
luvaldd
Purtill
Oversized
Replica
ScunNette
lavalid
Overlap
Pescuntion
ObjSDF
97
6016
71.8
52
PhyRRrao
Di-Rusco
97
Table 4: Quantitative Failure Rate Comparison: HoloScene achieves significantly lower geometry and physics failure rates compared reconstruction--un basclines, demonstrating more robust 3D reconstruction and physically plausible scene modeling.
4.2 Interactive Environment Applications
Real-Time Interactive Game With reconstructed environment, create real-time interactive game with Unreal Engine [2]. As Fig. shows, we build third-person game with the reconstructed texture meshes. The objects could be physically rearranged in the game world, and the game agent could also interact with the scene through realistic physics.
Interactive 3D Editing In simulation environment, could also achieve high-quality inter- active 3D editing by moving the object Gaussians with its underlying physical mesh geometry. In Fig. I demonstrate this by changing the location and orientation of the interactable chair.
Immersive Experience Recording We show our interactable reconstructed 3D objects with im- mersive experience recording. Given static RGB video of person manipulating object, aim to recover the object's 6l) pose and resimulate its motion in a virtual 3D) scene. We recover the camera pose with VGGT [69], adjust the predicted depth [54] to align with the virtual scene, and adopt Foundation Pose pose estimation. As shown in Fig. [ enable consistent replay of real- world interactions in virtual scenes while accurately recovering the object's pose from visual input.
73 for object tracking with our reconstructed 3D object for model-based 6D
Figure 5: Dynamic VFX Results. We augment the inferred interactive 3D scene with various visual effects such as dropping objects, adding animations, and fires.
To enhance immersion, we augment the scene with dynamic visual effects, including rigid body simulations, character animations, and particle effects. We adopt visual effeets from VFX [18] to overlay virtual content and shadows onto the image. As Fig, shows, we produce effects that blend naturally with the scene.
Conclusion & Limitation
We presented HoloScene, a novel interactive 3D modeling framework that uses scene graphs and energy-based optimization to reconstruct environments with realistic appearance, complete geometry, and interactive physical plausibility, achieving superior real-time rendering, geometric accuracy, and stable simulation. Limitations: HoloScene currently only handles videos of static indoor scenes; dynamic scenes and large outdoor environments remain challenging. Future work will focus on relightable reconstruction and extending support to articulated and deformable objects.
Acknowledgment: This project is supported by the Intel AI SRS gift and NSF Awards #2331878. #2340254, #2312102, #2414227, and #2404385. We greatly appreciate the NCSA for providing computing resources through Delta and Delta AI program.
10

---

References
[1] Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I,, Aleman, F.L., Almeida, D., Altenschmidt, J.,
Altman, S., Anadkat, S., et al.: Gpt-4 technical report. arXiv preprint arXiv:2303.0877 (2023)
[2] Amini, A., Wang, T.H., Gilitschenski, I., Schwarting. W. Liu, Z., Han, S., Karaman, S., Rus, D.: Vista 2.0: An open, data-driven simulator for multimodal sensing and policy learning for autonomous vehicles. In: ICRA (2022)
[3] Barron, J.T. Mildenhall, B., Tancik, M., Hedman, P., Martin- Brualla, R., Srinivasan, P.P.: Mip-nerf: multiscale representation for anti-aliasing neural radiance fields. In: ICCV (2021)
[4] Barron, J.T., Mildenhall, B., Verbin, D., Srinivasan, P.P.. Hedman, P:: Mip-nerf 360: Unbounded anti- anti-aliased neural radiance fields. In: CVPR (2022)
[5] Berger, M., Tagliasacchi, A., Seversky, L.M., Alliez, P., Levine, J.A., Sharf, A., Silva, C.T.: State of the art in surface reconstruction from point clouds. In: 35th Annual Conference of the European Association for Computer Graphics, Eurographics 2014-State of the Art Reports. The Eurographies Association (2014)
[6] Chen, B., Jiang, H., Liu, S., Gupta, S., Li, Y., Zhao, H., Wang, S.: Physgen3d: Crafting miniature interactive world from single image. arXiv preprint arXiv:2503..077 (2025)
[7] Chen, Y., Ni, J., Jiang, N., Zhang. Y., Zhu, Y., Huang, S.: Single- view 3D scene reconstruction with high-fidelity shape and texture. In: 2024 International Conference on 3D Vision (3DV). IEEE (2024)
[8] Chen, Y.. Rong, F., Duggal, S., Wang. S.. Yan. X., Manivasagam, S., Xue, S., Yumer, E., Urtasun, R.: Geosim: Realistic video simulation via geometry-awand composition for self-driving. In: CVPR (2021)
[9] Chen, Z.,, Walsman, A., Memmel, M., Mo, K., Fang, A., Vemuri, K., Wu, A., Fox, D., Gupta, A.: Urdformer:
A pipeline for constructing articulated simulation environments from real-world images. arXiv (2024)
[10] Dai, T., Wong, J., Jiang, Wang, C., Gokmen, C., Zhang, R., Wu, Fei-Fei, L.: Acde: Automated creation of digital cousins for robust policy learning, arXiv (2024)
[11] Dogaru, A., Özer, M., Egger, B.: Generalizable scene reconstruction divide and conquer from a single view. arXiv preprint arXiv:2404.03421 (2024)
[12] Epic Games: Unreal htap:neettt::/// httpppsswwwww.w..eeeeeniieeee..
[13] Feng, Y., Feng, X., Shang, Y., Jiang, Y., Yu, C., Zong, 7., Shao, T., Wu, H., Zhou, K., Jiang, C., al.: Gaussian splashing: Dynamic fluid synthesis with gaussian splatting. arXiv e-prints pp. arXiv-2401 (2024)
[14] Garcia, G.M., Abou Zeid, K., Schmidt, C., De Geus, D., Hermans, A., Leibe, B.: Fine-tuning image- conditional diffusion models is easier than you think. In: 2025 IEEE/CVF Winter Conference on Applica- tions of Computer Vision (WACV). 752-762. IEEE (2025)
[15] Guo, M., Wang, B., Ma, P., Zhang, T., Owens, C., Gan, C., Tenenbaum, J., He, K., Matusik, W.: Physically compatible 3d object modeling from single image. Advances in Neural Information Processing Systems 37. 119260-111999 19282 (2024)
[16] Guédon, A., Lepetit, V:: Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering (2023)
[17] Han, X.F., Laga, H., Bennamoun, M.: Image- based 3d object reconstruction: oftte-of-thh-art and trends in the deep learning era. IEEE transactions on pattern analysis and machine intelligence 43(5), (2019)
[18] Hsu, H.Y., Lin, Z.H., Zhai, A., Xia, H., Wang. S.: Autovfx: Physically realistic video editing from natural language instructions. arXiv preprint arXiv:2411.022 0.2994 (2024)
[19] Hu, S., Zhou, K., Li, K. Yu, L.,, Hong, L., Hu, T., Li, Z., Lee, G.H., Liu, Consistentnerf: Enhancing neural radiance fields with 3d consistency for sparse view synthesis. arXiv preprint XrX:::2305.1 1031 (2023)
[20] Huang, B., Yu, Z., Chen, A., Geiger, A., Gao, S.: 2d gaussian splatting for geometrically accurate radiance fields. In: SIGGRAPH. Association for Computing Machinery (2024)
[21] Jiang, H., Hsu, H.Y., Zhang, K. Yu. H.N., Wang. S., Li, Y:: Phystwin: Physics- informed reconstruction and simulation of deformable objects from videos. arXiv preprint arXiv:2503. 17973 (2025)
11

---

[22] Jiang. Y., Yu, C., Xie. T., Li, X., Feng. Y.. Wang, H., Li, M., Lau, H., Gao, F. Yang. Y., et al.: Vr-gs: A physical aynaeics-aware interactive gaussian splatting system in virtual reality. In: ACM SIGGRAPHI 2024 Conference Papers. pp. 1-1 (2024)
[23] Kerbl. B., Kopanas, G., Leimkühler. T., Drettakis, G.: 3d gaussian splatting for real-time radiance field rendering. TOG (2023)
[24] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L.,, Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., et al.: Segment anything. In: ICCV (2023)
[25] Kwak, M.S., Song, J., Kim, S.: Geconerf: Few-shot neural radiance fields via geometric consistency. arXiv preprint arXiv:2301. (2023)
[26] Li, C., Xia, Martin-Martin, R., Lingelhach, M., Srivastava, S., Shen, B., Vainio, K., Gokmen, C., Dharan, G., Jain, T., et al.: igibson 2.0: Object-centric simulation for robot learning of everyday household tasks. arXiv preprint arXiv:2108.00 100.02222 (2021)
[27] Li, X., Li, J. Zhang, Z.. Zhang, R., Jia, F., Wang. T., Fan, H., Tseng, K.K., Wang, R.: Robogsim: real2sim2real robotic gaussian splatting simulator. arXiv preprint arXiv:2411. 1839 (2024)
[28] Li, Y., Lin, Z.H., Forsyth, D., Huang. J.B., Wang, S.: Climatenerf: Physically-hased neural rendering for extreme climate synthesis. arXiv (2022)
[29] Li, Z., Müller, T., Evans, A., Taylor, R.H., Unberath, M., Liu, M.Y., Lin, C.H.: Neuralangelo: High-fidelity neural surface reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 8456-8465 (2023)
[30] Li, Z., Lyu, X., Ding, Y., Wang, M., Liao, Y., Liu, Y.: Rico: Regularizing the unobservable for indoor compositional reconstruction. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 17761-17771 (2023)
[31] Lin, Z.H., Huang. J.B., Li, Z., Dong, 7., Richardt, C., Li, T., Zollhöfer, M., Kopf, J., Wang. S., Kim, C.: Iris: Inverse rendering of indoor scenes from low dynamic range images. arXiv preprint 2Xxv:2401.12977 (2024)
[32] Lin, Z.H., Liu, B., Chen, Y.T., Forsyth, D., Huang, J.B., Bhattad, A., Wang, S.: Urbanir: Large-scale urban scene inverse rendering from a single video (2023)
[33] Ling, L.,, Lin, C.H., Lin, T.Y., Ding, Y., Zeng, Y., Sheng, Y., Ge, Y., Liu, M.Y., Bera, A., Li, Z.: Scenethesis: A language and vision agentic framework for 3d scene generation. arXiv preprint arXiv:2505 028366 (2025)
[34] Liu, J.Y., Chen, Y., Yang, Z.,, Wang, J., Manivasagam, S., Urtasun, R.: Real-time neural rasterization for large scenes. In: Proceedings of the IEFF/CV International Conference on Computer Vision. pp. 1416-8427 (2023)
[35] Liu, R., Wu, R., Van Hoorick, B., Tokmakov, P., Zakharov, S., Vondrick, C.: Zero-I-to-3: Zero-shot one image to 3d object (2023)
[36] Liu, S., Ren, Z., Gupta, S., Wang, S.: Physgen: Rigid-body physics-grounded imago-to-video generation. In: European Conference on Computer Vision. pp. 360-378. Springer (2024)
[37] Liu, Y. Jia, B., Lu, R., Ni, J., Zhu, S.C., Huang, S.: Building interactable replicas complex articulated objects via gaussian splatting, arXiv preprint ar Xiv:222500. 19459 (2025)
[38] Liu, Z., Zhou, G., Hc, J., Marcucei, T., Li, F.F. Wu, J., Li, Y.: Model-based control with sparse neural dynamics. Advances in Neural Information Processing Systems 36, 0280-6296 (2023)
[39] Long, X., Guo, Y.C., Lin, C., Liu, Y., Dou. Z., Liu, L., Ma. Y., Zhang. S.H., Habermann, M.. Theobalt, C., et al.: Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310. 1500000008000 (2023)
[40] Lu, F. Xu, Y., Chen, G., Li, H., Lin, K. K.Y., Jiang, C.: Urban radiance field representation with deformable neural mesh primitives (2023)
[41] Lu, R., Chen, Y., Ni, J., Jia, B., Liu, Y., Wan, D., Zeng, G., Huang, S.: Movis: Enhancing multi-object novel view synthesis for indoor scenes. arXiv preprint arXiv:2412. 1457 (2024)
[42] Luo, R., Geng, H., Deng, C., Li, P., Wang, Z. Jia, B., Guibas, L., Huang, S.: Physpart: Physically plausible part completion for interactable objects. arXiv preprint arXivv2408. 137244 (2024)
12

---

[43] Ma, S., Du, W., Yu. C., Jiang. Y., Zong, Z., Xie. T., Chen, Y. Yang. Y., Han, X., Jiang, C.: Grip: A general robotic incremental potential contact simulation dataset for unified deformable-rigid coupled grasping. arXiv preprint arXiv:2503.00022 (2025)
[44] Manivasagam, S., Wang, S., Wong, K., Zeng. W., Sazanovich, M.. Tan, S., Yang, B., Ma, W.C., Urtasun, R:: Lidarsim: Realistic lidar simulation by leveraging the real world. In: CVPR (2020)
[45] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. In: ECCV (2020)
[46] Mildenhall, B., Srinivasan, P.P., Tancik, M., Barron, J.T., Ramamoorthi, R., Ng, R.: Nerf: Representing scenes as neural radiance fields for view synthesis. ACM Communications (2021)
[47] Mittal, M., Yu, C., Yu. Q. Liu, J., Rudin, N.. Hoeller, D., Yuan, J.L., Singh, R., Guo. Y., Mazhar, H., Mandlekar, A., Babich, B., State, G., Hutter, M., Garg, A.; Orbit: A unified simulation framework for interactive robot learning environments. IEEE Robotics and Automation Letters 8(6), 3740-3744 (2023). https://doi.org/100. 1099LRA
[48] Ni, J., Chen, Y., Jing, B., Jiang, N., Wang, B., Dai, B., Li, P., Zhu, Y., Zhu, S.C., Huang, S.: Phyrecon: Physically plausible neural scene reconstruction. arXiv preprint arXiv:2404, 16666 (2024)
[49] Ni, J., Liu, Y., Lu, R., Zhou, Z., Zhu, S.C., Chen, Y., Huang, S.: Decompositional neural scene reconstruction with generative diffusion prior (2025)
[50] Ni, J. Liu, Y., Lu. R., Zhou, Z.. Zhu, S.C., Chen, Y., Huang, S.; Decompositional neural scene reconstruction with generative diffusion prior. arXiv preprint arXiv:2503. 1 14830 (2025)
[51] Occhsle, M., Peng. S., Geiger, A:: Unisurf: Unifying neural implicit surfaces and radiance fields for multi-view reconstruction. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 5589-5599 (2021)
[52] Özsoy, E.. Czempiel. T.. Ömek. E.P., Eck, U.. Tombari. F., Navab, Holistic or domain modeling: a semantic scene graph approach. International Journal of Computer Assisted Radiology and Surgery 19(5), 791-799 (2024)
[53] Park, J.J., Florence, P., Straub, J., Newcombe, R., Lovegrove, S.: Deepsdf: Learning continuous signed distance functions for shape representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 165-174 (2019)
[54] Piccinelli, L., Sakaridis, C., Yang, Y.H., Segu, M., Li. S., Abbeloos, W., Van Gool, L.: Unidepthv2: Universal monocular metric depth estimation made simpler. arXiv preprint arXiv:2502.20110 (2025)
[55] Poole, B., Jain, A., Barron, J.T., Mildenhall, B.: Dreamfusion: Text-to-3D using 2D diffusion. arXiv preprint arXiv:2209. 14988 (2022)
[56] Powell, M.J.; An efficient method for finding the minimum of a function of several variables without calculating derivatives. The computer journal 7(2), 155-162 (1964)
[57] Pun, A., Sun, G., Wang, J., Chen, Y., Yang. Z., Manivasagam, S., Ma, W.C., Urtasun, R.: Lightsim: Neural lighting simulation for urban scenes (2023), httpe://arxiv org/ 20522/20//3110654
[58] Pun, A., Sun, G., Wang, J., Chen, Y., Yang, Z., Manivasagam, S., Ma. W.C., Urtasun, R.: Neural lighting simulation for urban scenes. NeurIPS (2024)
[59] Qian, S., Fouhey, D.F. Understanding 3D object interaction from single image. In: ICCV (2023)
[60] Ren, T., Liu, S., Zeng, A., Lin, J., Li, K., Cao, H., Chen, J., Huang, X., Chen, Y., Yan, F,, Zeng, Z., Zhang, H., Li, F., Yang, J., H., Jiang, Q., Zhang, L.: Grounded sam: Assembling open-world models for diverse visual tasks (2024), https::/ // //tp://aa/vvv////////a org/ / abs//401..2/./ 14159
[61] Sco, S., Han, D., Chang, Y., Kwak, N.: Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs. In: Proceedings of the IRFE/CVF Conference on Computer Vision and Pattern Recognition. pp. 20659-20666 (2023)
[62] Somraj. N., Soundararajan, R.: Vip-nerf: Visibility prior for sparse input neural radiance fields. In: ACM SIGGRAPH 2023 Conference Proceedings. pp. 1-11 (2023)
[63] Son, S., Qiao, Y.L., Sewall, J., Lin, M.C.: Differentiable hybrid traffic simulation. TOG (2022)
13

---

[64] Straub, J.. Whelan, T., Ma, L., Chen, Y., Wijmans, E.. Green, S., Engel, JJJ., Mur-Artal, R., Ren, C., Verma, S., Clarkson, A.. Yan, M., Budge, B., Yan, Y., Pan, X., Yon, J., Zou, Y., Leon, K., Carter. N., Briales, J., Gillingham, T., Mueggler, E., Pesqueira, L., Savva, M., Batra, D., Strasdat, H.M., Nardi, R.D., Goesele. M., Lovegrove, S., Newcombe, R.; The Replica datasct: A digital replica of indoor spaces, arXiv preprint arXiv:1906.0553 (2019)
[65] Sucar, E., Liu, S., Ortiz, J., Davison, A.J.: imap: Implicit mapping and positioning real-time. In: Proceedings of the IEEE/CVF international conference on computer vision. pp.
(2021)
[66] Suvorov, R., Logacheva, E., Mashikhin, A., Remizova, A., Ashukha, A., Silvestrov, A., Kong. N., Goka, H., Park, K., Lempitsky, V:: Resolution-rouust large mask inpainting with fourier convolutions. arXiv preprint arXiv:2109.07161 (2021)
[67] Torne, M., Simeonov, A., Li, Z., Chan, A., Chen, T., Gupta, A., Agrawal, P.: Reconciling reality through simulation: A real-to-sim-to-rall approach for robust manipulation. arXiv (2024)
[68] Wang, H., Li, M.: A new era of indoor scene reconstruction: A survey. IEEE Access (2024)
[69] Wang, J., Chen, M., Karaev, N., Vedaldi, A., Rupprecht, C., Novotny, D.: Vggt: Visual geometry grounded transformer. arXiv preprint arXiv:2503. 1651 (2025)
[70] Wang, L., Guo, R., Vuong, Q. Qin, Y., Su, H., Christensen, H.: A real2sim2real method for robust object grasping with neural surface reconstruction. In; 2023 IEEE 19th Internatoonal Conference on Automation Science and Engineering (CASE). pp. 1-8. IEEE (2023)
[71] Wang, P. Liu, L.,, Liu, Y., Theohalt, C., Komura, T., Wang, W.: Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. arXiv (2021)
[72] Wang, Y., Qiu, X., Liu,J. Chen, Z., Cai, J., Wang. Y., Wang. T.H., Xian, Z., Gan, C.: Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting. Advances in Neural Information Processing Systems 37, 67575 67603 (2024)
[73] Wen, B., Yang, W., Kautz. J., Birchfield, S.: Foundationpose: Unified 6d pose estimation and tracking of novel objects. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 17868-1788 (2024)
[74] Wu, Q. Liu, X., Chen, Y., Li. K.. Zheng, C., Cai, J.. Zheng, J.: Object- neural implicit surfaces. In: European Conference on Computer Vision. pp. 197-213. Springer (2022)
[75] Wu, Q. Wang, K., Li, K., Zheng, J., Cai, J.: Objectsdf++: Improved pojjectcommositionalll neural implicit surfaces. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 21764 21774 (2023)
[76] Wu, T., Zheng, C., Guan, F., Vedaldi, A., Cham, T.J.: Amodal3r: Amodal 3d reconstruction from occluded 2d images. arXiv preprint arXiv:2503. 13439 (2025)
[77] Xia, H., Lin, Z.H., Ma, W.C., Wang. S.: Video2game: Real-time interactive realistic and hrowser-compatible environment from a single video. In: CVPR (2024)
[78] Xia, H., Su, E., Memmel, M., Jain, A., Yu, R., Mbiziwo-Tian Tiapo, N., Farhadi, A., Gupta, A., Wang, S., Ma, W.C.: Drawer: Digital reconstruction and articulation with environment realism (2025), litpps
/ /aaxiiv.oo warxiv.or//bba. bbrgg/bba//00 2504. 15278
[79] Xiang, J.. Lv, Z., Xu, S., Deng, Y., Wang, R., Zhang, B., Chen, D., Tong, X., Yang, J.: Structured 3d latents for scalable and versatile 3d generation. arXiv preprint arXiv:2412.01508 (2024)
[80] Xie, T., Zong, 7., Qiu, Y., Li, X., Feng, Y., Yang. Y., Jiang, C.: Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4389 4398 (2024)
[81] Xiong, Y., Ma. Wei-Chiu Wang. J., Urtasun, R.: Ultralidar: Learning compact representations for lidar completion and generation. CVPR (2023)
[82] Yang, J.. Ivanovic, B., Litany, O., Weng. X., Kim, S.W., Li, B., Che. T., Xu, D., Fidler, S., Pavone, M., Wang, Y.: Emernerf: Emergent spatial-temprra scene decomposition via self self-supervision. In: ICLR (2024)
[83] Yang. Y., Sun, F.Y., Weihs, L., VanderBilt, E., Herrasti, A., Han, W., Wu, J., Haber. N., Krishna, R., Liu, L.. et al.: Holodeck: Language guided generation of 3d embodied ai environments. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16227-16233 (2024)
14

---

[84] Yang, Z., Chen, Y. Wang, J., Manivasagam, S., Ma. W.C., Yang, AJ.. Urtasun, R.: Unisim: A neural closed-loop sensor simulator. CVPR (2023)
[85] Yang. Z., Chai, Y., Anguelov, D., Zhou, Y., Sun, P., Erhan, D., Rafferty, S., Kretzschmar, H.: Surfelgan: Synthesizing realistic sensor data for autonomous driving. In: CVPR (2020)
[86] Yao, K., Zhang, L., Yan, X., Zeng, Y., Zhang, Q. Xu, L.,, Yang. W. Gu, J., Yu, J.: Cast: Component-aligned 3D reconstruction from rgh image. arXiv preprint urXiv:2502.1255 (2025)
[87] Yariv, I.,, Gu, J., Kasten, Y., Lipman, Y: Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems 34, 4805-4815 (2021)
[88] Yariv, L.,, Hedman, P., Reiser, C., Verbin, D., Srinivasan, P.P., Szeliski, R., Barron, J.T., Mildenhall, B.:
Bakedsdf: Meshing neural sdfs for real-time view synthesis. In: SIGGRAPH Conference (2023)
889] Yeshwanth, C.,, Liu, Y.C., Nicßner, M., Dai, A.: Scannet++: A high-fidelity dataset of 3D indoor scenes. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 12-22 (2023)
[90] Yu, Z.,, Chen, A., Huang, B., Sattler, T., Geiger, A.: Mip-splatting: Alias-free 3d gaussian splatting (2024)
[91] Yu, Z., Peng, S., Niemeyer, M., Sattler, T., Geiger, A.: Monosdf: Exploring monocular geometric cues for neural implicit surface reconstruction. arXiv (2022)
[92] Yu, Z., Sattler, T., Geiger, A.: Gaussian opacity fields: Efficient and compact surface reconstruction in unbounded scenes. arXiv (2024)
[93] Yuan, Y., Bleier, M., Nuchter, A.: Scenefactory; A workflow- centric and unified framework for incremental scene modeling. IEEE Transactions on Robotics (2025)
[94] Zhai, A.J., Shen, Y., Chen, E.Y., Wang, G.X., Wang. X., Wang, S., Guan, K., Wang, S.: Physical property understanding from language-embedded feature fields (2024)
[95] Zhang, K., Li, B., Hauser, K., Li, Y.: Particle-grid neural dynamics for learning deformable object models from rgb-d videos. In: Proceedings of Roboties: Science and Systems (RSS) (2025)
[96] Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unrcasonable effectiveness of deep features perceptual metrie. In: CVPR (2018)
[97] Zhao, H.. Wang. H., Zhao, X., Wang, H., Wu. Z.. Long, C.. Zou. H.: Automated 3d physical simulation of open-world scene with gaussian splatting. arXiv preprint arXiv:241 12789 (2024)
[98] Zyrianov, V., Che. H., Liu, Z., Wang, S.: Lidardm: Generative lidar simulation in a generated world (2024),
petps://arxite org/ abss/204 02903
[99] Zyrianov, V., Zhu, X., Wang, S.: Learning to generate realistic lidar point clouds. In: ECCV (2022)
15

---

Appendix
A
More Interactive Environment Application Results
Real-Time Interactive Game With the reconstruction results of HoloScene, we could create an interactive game in Unreal Engine [2]. which supports real-time interactions with our created assets. We import all the objects and the background texture mesh into the game engine and create a third- person game, where the agent can freely explore, navigate, jump, and interact with the objects in the scene. Thanks the complete geometry and appearance, as well as the great physical plausibility of HoloScene, the objects could remain stable in the game. Also, the movements of objects could follow physical rules when the objects interact. The Fig. [ shows the game screenshots, and more videos could be found in the supp. material as well.
Interactive 3D Editing With each object's appearance and geometry reconstructed, we support interactive 3D scene editing by allowing users to manipulate individual Gaussians through a control panel. Objects can be translated along the 2D xy-plane and rotated around the vertical (2) axis. As illustrated in Fig. 2 the reconstructed swivel chair is repositioned freely and rendered realistically, resulting in coherent and intuitive scene modifications.
Immersive Experience Recording We demonstrate the usage of the reconstructed 3D objects from HoloScene for immersive experience recording. Given statie RGB video of person manipulating an object, our goal is to recover the object's 6D pose and resimulate its motion within a virtual 3D scene. We use the first frame of the video the target view for camera calibration. To determine the camera pose, we first estimate relative transformations between the target and nearby training views using VGGT 69). We then solve least-squares problem with Powell's method (56) to derive global transformation that aligns the VGGT-predicted poses with the ground-truth poses of the training views. This transformation is applied register the target view within the training coordinate system. We use UniDepth V2 [54] to predict the depth map for each frame and compute a scale ratio between the predicted monocular depth and depth rendered from the 3D Gaussians. This ratio adjusts the predicted depth to align accurately within the virtual scene. For object tracking, we adopt Foundation (73). using our high-quality reconstructed 3D asset as a CAD model for model-based 6D pose estimation. The recovered pose sequence is then applied to the 3D object and its 3D Gaussians for final rendering. As illustrated in Fig. 3 we accurately recover the object's pose from visual input, enabling consistent replay of real-world interactions in virtual scenes.
Dynamic Visual Effects We showcase an immersive application by augmenting the reconstructed scenes with dynamic visual effects. These include rigid body simulations, 3D) character animations, and particle-based effects, all spatially grounded within the virtual scene. We adopt multiple visual effects created by Auto VFX 18 to simulate virtual content and overlay it, along with shadows, onto produce effects that blend naturally with the scene, enhancing
the input image. As shown in Fig. both visual richness and physical realism.
Figure I: More Interactive Game Results: We show more screenshots of the interactive game built by Unreal Engine here. The agent can run, jump, and interact with all the objects in the game.
B More Physical Simulation Results
In this section, we will show more physical simulation results of Holoscene and its comparison with the bascline methods 75 48 49 In Fig. 5 and Fig. 6 we show the comparisons of our method and

---

Figure 2: More Interactive 3D Editing Results: The user could interact with the object in the scene, change the location and orientation of the object the control panel, and attain the edited scene.
Pose D (lan))
Pose
Li
Pose
Pose
Pose (lan)
Pose
Plaats Late
Figure 3: More Immersive Experience Recording Results: Here show results of the immersive experience recording of two objects the scene. For each image pair, the original recording is shown the left, and the simulated object rendering with Gaussians is the right. The estimated object 6D poses are marked with oriented bounding boxes.
the baselines in the Scannet++ 89 dataset. In Fig. 7 we show the comparisons the Replica 664 dataset. In Fig- [ show the comparisons in the iGibson (26) dataset. From the results, observe that HoToScene can achieve the best physical stability and plausibility, as well as realistic rendering quality.
C Method Details
Structured Tree Search When we perform the breadth-firg first traversal on the scene tree structure, for each node, will end up choosing the reconstructed object candidate with the lowest physical energy. Then we will further adjust the state of the chosen candidate object to ensure its stability. Specifically. we will regularize the SDF value and the mesh vertex locations to prevent the interpenetration of objects. Before applying marching cube to the instance SDF, we will first prune the SDF value of the points if those points fall into the boundary of its parent object or nearby objects. After we attain the mesh with marching cube, we will perform collision test of the reconstructed object mesh with its parent object and the nearby objects. And if there exist collisions, we will dynamically optimize the locations of the vertices to avoid the collisions. After the optimization, the contact surfaces of

---

HILL
Figure 4: More Dynamic Visual Effect Results: We show more results of dynamic visual effects, including animations, dropping objects, and setting fires.
objects will be regularized and avoid inter-object collisions, so that the stability of the scene can be further improved.
D
Experiment Details
D.1 Datasets
We adopt three datasets we mentioned in our main paper: Replica [64], Scannet++ [89] and iGibson [26). Replica:
The original Replica dataset only provides the texture mesh with instance annotations, and no existing videos with camera trajectories are provided. We manually design dense camera trajectory several hundred frames for each scene, and render images with a size of (512, 512). Scannet++: The scannet++ dataset provides multi-view posed images and the aligned 3D scan with instance annotation. However, limited by the quality of the scan, the instance mask derived from the annotation is not ideal. We adopt [24] to get cleaner instance masks instead. For the image resolution, we downsample the original image by 2. iGibson: The iGibson dataset provides the complete geometry and appearance of every object and the background scene, but without provided camera trajectories. We manually design dense camera trajectory of several hundred frames for each scene, and render images with size of (512, 512).
D.2 Baselines
We adopt the open-source codes the baselines and adapt them for our benchmark. Our baselines are ObjectSDF++ (75), PhyRecon [48), and DP-Recon [49]. Now we will elaborate the details of them.
ObjectSDF++:
The original codebase only supports their rendering images from Replica 64 where each image has size of (384, 384). In order make adaptable other image resolutions, we implement a new data loader in the code.
RecRe::n: Similar to ObjectSDF, the original code base only supports their rendering images from Replica (64 and different resolution resized images from Scannet++ [89]. For the adaption to other image resolutions, implement new data loader in the code. The PhyRecon has a final stage

---

OBDDD
Ref Image
PhyRecca
Genmetry
DP-Rocon
DP-Recon
Physies
With Physics (Intermediate)
With Physics (Final)
Figure 5: More Physical Simulation Results in Scannet++ Dataset Scene 1: We show the geometry and appearance of each object in the scene with physical simulation, from no physics state, intermedi- state with physics, and final state with physics.
of applying physical loss to every object. But may happen that some objects fail to reconstruct themselves, the physical loss can not be applied to those objects. In this case, we skip those objects.
DP-Recon: Similar the baselines above, implement new data loader to adapt the code our dataset. DP-Recon has the geometry stage and the following appearance stage. We follow the officially released code run those experiments. DP-Recon needs generate the object bounding boxes in the running process. But DP-Recon overlooks the possibility that some objects may fail be reconstructed and disappear. So in this case, largest possible bounding box in the scene to fix the issue. We evaluate the appearance, geometry, and physics based on the final texture mesh results from the second stage of DP-Recon.
the bounding box of the object as the
D.3 The inference details of HoloScene
The inference stage HoloScene is composed three stages after generating scene graph in stage 0, as we mentioned in the main paper. In stage of the gradient-basse optimization, we follow the previous instance-level SDF [75] to set the hyperparameters and train for 200k iterations similarly. In stage 2 of the sampling-he based based optimization, we adopt the Isaac Sim [4] as the physical simulator,

---

ObjSDE
Ref Image
PbyRecca
Genmetry
DP-Rocon
Appearance DP-Recon
Physies
With Physics (Intermediate)
With Physics (Final)
Figure 6: More Physical Simulation Results in Scannet++ Dataset Scene 2: We show the geometry and appearance of each object in the scene with physical simulation, from no physics state, intermedi- state with physics, and final state with physics.
and set the physical parameters in the simulation default. In the sampling with generative models, we randomly choose the seeds and generate at most five candidates for each object. In stage of the gradient-based refinement, we refine the trained Gaussians globally. We optimize the Gaussians for 30k iterations following [23], and in each iteration, we also use generated images to optimize the instance Gaussians as well. For other learning parameters, they are set default our experiments.

---

ObJSDK
Ref Image
Pby PhyyRooo
Geometry
DR-Rocon
Appearance
Physies
Physics (Intermediate))
With Physicx (Final)
Figure 7: More Physical Simulation Results in Replica Dataset: We show the geometry and appearance of each object in the scene with physical simulation, from no physics state, intermediate state with physics, and final state with physics.

---

Ref Imuge
PhyyRoco
Genmetry
DP-Recon
Physies
Physicx (Intermechate)
With Physics (Final)
Figure 8: More Physical Simulation Results in iGibson Dataset: We show the geometry and appearance of each object in the scene with physical simulation, from no physics state, intermediate state with physics, and final state with physics. De-Recon fails to reconstruct the complete background mesh, leading to object falling in the figure.

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our claims match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in our paper.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper doesn't I focus on the theoretical derivation.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We fully disclose method and the experiment settings in the paper.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We would open-source the data and code of the paper.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We elaborate the experiment settings detail in our paper.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We conduct multiple sets of experiments alleviate the statistical significance of the experiments.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Ys]]
Justification: We provide the training details in the supplementary material.
9. Code of ethics

---

Question: Does the research conducted in the paper conform, in every respect. with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: We have read and conducted the paper conform, in every respect, with the NeurlPS Code of Ethics.
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the potential impacts in the paper.
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: The paper has no such risks.
12. Licenses for existing assets
Are the creators or original owners of assets (e-g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The creators or original owners of assets used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects. 15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development this research does not involve LLMs as any important, original, or non-standard components.


Reviews:
[{"review_id": "Aq4btBZLLQ", "reviewer": "Reviewer_nTyK", "summary": "The paper introduces HoloScene, a method for generating interactive, simulation-ready 3D scenes from videos of static environments. The approach combines generative sampling and scene graph-based tree search with energy-based optimization to ensure realistic appearance, geometric completeness, and physical plausibility. The authors evaluate their method on multiple datasets (iGibson, ScanNet++, and Replica), and compare against several object- and scene-level reconstruction baselines (e.g., PhyRecon, ObjSDF++, DP-Recon), demonstrating superior reconstruction quality.", "questions": "1.\tCould the authors show example input videos (how dense/sparse they are)? What is the typical length? Also, how does the method work if given unstructured image collections instead of a video?\n2.\tWhy is the reconstruction error lower at object level for iGibson, but not at scene level? Some insight would be helpful.\n3.\tBalancing Training Samples (L268): The idea of balancing samples across all instances—especially to improve recovery of small objects—seems important, but it is not sufficiently explained.\n4.\tOcclusion Inpainting (L164): How are occluded regions detected for inpainting with LaMa? Is this step performed after reconstructing with Wonder3D?\n5.\tThere appears to be some blurring in the rendered appearance compared to standard 3D Gaussian Splatting (3DGS). What limits the realism here, and how might it be improved?\nThese could help to improve the exposition (and my score :P)", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "The work is quite impressive. For example, it is \n\n1.\tComprehensive: The method considers most aspects of 3D world, addressing both scene-level and object-level reconstruction, producing physically plausible and high-quality outputs while supporting simulation and real-time rendering.\n\n2.\tNovelty: The proposed divide-and-conquer strategy, which integrates sampling-based tree-structured search with gradient-based refinement, is elegant and effective.\n\n3.\tStrong Results: HoloScene achieves superior performance across geometry accuracy, real-time rendering, and physical plausibility for downstream tasks, with broad evaluations across diverse datasets.\n\n4.\tPresentation Quality: The paper is well-written and well-structured, making the core ideas and technical contributions easy to follow.\n\nThere are no major weaknesses, but I have a few minor concerns and suggestions for clarification: The paper employs multiple geometry representations (meshes, SDFs, and Gaussians), but the motivation and integration of these components is not clearly explained: Why is an SDF representation needed, given that meshes are required for simulation and Gaussians for rendering? How are SDFs represented—globally per object or truncated SDF? Are the SDF and mesh reconstructions synchronized (e.g., via Marching Cubes)? Overall, the SDF/Mesh/Gaussian is a bit sloppily written, or at least missing motivation for SDF. It’s also unclear if the reconstruction geometry is evaluated with mesh/SDF.", "was_revised": true, "final_justification": "I had minor comments and questions, which the authors addressed well. I carefully considered the rebuttal and other reviews/rebuttals. Overall, the method is quite comprehensive, and I believe that technical contributions and evaluations are sufficient for publication.", "rebuttal": "We thank the reviewer for appreciating the elegant and comprehensive design and superior performance of the simulation-ready environment. \n\nWe now address the concerns.\n\n**Why is an SDF representation needed?**  \nDirectly optimizing surface meshes from video is challenging due to their complex and discrete topology, especially in large indoor scenes. Neural SDFs provide a flexible, continuous representation that simplifies optimization and has demonstrated superior geometric quality [6, 7]. Once optimized, we extract meshes via marching cubes for simulation and attach Gaussians for high-quality rendering. This design balances reconstruction accuracy, topological flexibility, and downstream usability.\n\n**Are SDFs and meshes synchronized?**  \n*Yes*. Meshes are extracted from the SDF using marching cubes (L199–L206), ensuring they are fully synchronized.\n\n**Is geometry evaluated with meshes or SDFs?**  \nIn Table 2, we evaluate geometry quality using the *final reconstructed mesh*, following standard protocols used by our baselines. This ensures fair comparison across diverse approaches and accurately reflects downstream performance in simulation and other mesh-dependent tasks.\n\n**Could you show example input videos (how dense/sparse they are)?**  \nYes! We will include example input videos in the supplementary video and project website to illustrate capture density. Real-world inputs from ScanNet++ [2] are 1–2 minute sequences at 30 FPS with smooth handheld trajectories. Our synthetic datasets (Replica and iGibson) follow the same setup with comparable frame density and motion patterns.\n\n**What is the typical length?**  \nThe table below summarizes the average number of frames across different datasets. Our collected videos follow the capture speed and density of ScanNet++ [2]. Typical sequence lengths are several hundred frames for small indoor scenes (Replica, ScanNet++) and over a thousand frames for larger scenes (iGibson). For data we collected ourselves, we used handheld RGB cameras with smooth trajectories at 30 FPS, consistent with ScanNet++ practices. HoloScene is designed to handle input videos of varying lengths with ease. For additional statistics and visualizations, please refer to the ScanNet++ website.\n\n| Dataset | Replica[1] | Scannet++[2] |  iGibson[3]   |\n| -------- | -------- | -------- | --- |\n| Average frame #     | 638     | 365     |   1626  |\n\n**Can HoloScene handle unstructured image collections?**  \nYes! HoloScene supports unstructured image collections as input. The main difference lies in scene graph construction (L193–198): for videos, we sample keyframes and estimate relationships using a VLM, while for unstructured image collections, we can compute pairwise camera distances and incrementally build the scene graph. The remaining inference pipeline remains unchanged (L198).\n\n**Why is the reconstruction error at object level lower than scene level on iGibson?**  \nThis difference is due to scale and sampling resolution. Our evaluation follows standard geometry protocols [8, 9], which compute metrics by sampling points on surface geometry. At the object level, denser sampling within smaller regions increases the likelihood of finding closer correspondences, leading to lower errors. In contrast, scene-level evaluations span larger and more complex spaces, resulting in higher average error. Therefore, object- and scene-level errors are not directly comparable. That said, within each setting, the relative rankings across baselines still reliably reflect reconstruction quality.\n\n**Explanation of the idea of balancing samples across instances:**  \nIn Stage 1 (L199–L206), we sample an equal number of pixels for each instance in every optimization iteration. This ensures that smaller objects receive sufficient attention during training and are not overshadowed by larger instances (see L203 and L268).\n\n**How is occlusion inpainting done?**  \nWe first obtain reconstructed geometry for each object after Stage 1 inference (L199). Then, LaMa takes the rendered instance masks as input to inpaint occluded regions (L164). The inpainted images are subsequently passed to Wonder3D to generate novel views for invisible regions.\n\n**Question about rendering quality and potential improvements:**  \nWe adopt the Gaussian-on-Mesh (GoM) approach from DRAWER [4]. Unlike standard 3D Gaussian Splatting, GoM constrains each Gaussian’s position, orientation, and scale based on the underlying mesh. While this better supports physical simulation and improves scene editability, it reduces the flexibility of Gaussian optimization compared to unconstrained 3DGS, which often trades geometry accuracy for rendering quality. This trade-off explains the slightly blurrier appearance, as also reflected in the rendering metrics in Table 2. Similar limitations are observed in other GoM-based methods such as DRAWER [4] and GaMeS [5]. Rendering quality can be improved by densifying the Gaussians or relaxing mesh constraints, though this may reduce the alignment between appearance and geometry—potentially harming simulation fidelity.\n\n[1] Straub, Julian, et al. \"The replica dataset: A digital replica of indoor spaces.\" arXiv preprint arXiv:1906.05797 (2019).\n\n[2] Yeshwanth, Chandan, et al. \"Scannet++: A high-fidelity dataset of 3d indoor scenes.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Li, Chengshu, et al. \"igibson 2.0: Object-centric simulation for robot learning of everyday household tasks.\" arXiv preprint arXiv:2108.03272 (2021).\n\n[4] Xia, Hongchi, et al. \"Drawer: Digital reconstruction and articulation with environment realism.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[5] Waczyńska, Joanna, et al. \"Games: Mesh-based adapting and modification of gaussian splatting.\" arXiv preprint arXiv:2402.01459 (2024).\n\n[6] Yariv, Lior, et al. \"Bakedsdf: Meshing neural sdfs for real-time view synthesis.\" ACM SIGGRAPH 2023 conference proceedings. 2023.\n\n[7] Li, Zhaoshuo, et al. \"Neuralangelo: High-fidelity neural surface reconstruction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[8] Wu, Qianyi, et al. \"Objectsdf++: Improved object-compositional neural implicit surfaces.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[9] Ni, Junfeng, et al. \"Decompositional neural scene reconstruction with generative diffusion prior.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025."}, {"review_id": "EEOyoltmPF", "reviewer": "Reviewer_x1eu", "summary": "The study (HoloScene) presents a novel framework for reconstructing interactive, physically plausible 3D scenes from a single RGB video. HoloScene represents a scene as a structured 3D scene graph, where each node encodes object geometry (via neural SDFs and meshes), appearance (using Gaussian splats), and physical properties (mass, friction, etc.). Edges in the graph capture inter-object relationships (e.g., support or contact), enabling hierarchical and physically grounded interactions. The reconstruction is formulated as an energy-based optimization problem balancing observation fidelity (RGB, depth, masks), shape completeness, geometric consistency, and physical stability. The authors propose a three-stage pipeline: Initializing via gradient-based optimization to match observed inputs; Then diverse object completions are generated using a diffusion model (Wonder3D) and combined via a physics-aware search for plausible layouts; Finally enhance visual realism and ensure consistency.", "questions": "Please refer to the weakness comments.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**:\n\n***Unified Framework***: HoloScene integrates geometry reconstruction, appearance modeling, and physical simulation into a single coherent pipeline, enabling interactive and simulation-ready 3D scenes from just a video.\n\n***Energy-based Optimization***: The combination of diffusion-based shape completion with physics-guided tree search is novel to ensures visual quality and physical plausibility.\n\n**Weaknesses**:\n\n***Limited Literature Review***: The paper lacks a thorough discussion of related works in interactive 3D scene generation. Several recent generation-based methods—such as Holodeck, Digital Cousins, and Scenethesis—as well as reconstruction-based approaches like DeepPriorAssembly and Architect, have addressed similar goals. Many of these works consider physical plausibility, such as collision avoidance or stability constraints (e.g., Scenethesis considers both). A more comprehensive comparison would help position HoloScene more clearly within the landscape of prior research and highlight its unique contributions.\n\n***Insufficient Baseline Comparisons***: The experimental evaluation relies on relatively outdated baselines, such as ObjSDF++ (2023), which has already been compared with PhyRecon and DP-Recon. More recent state-of-the-art methods, especially those mentioned above, should be included in the comparisons or at least discussed to strengthen the empirical claims.\n\n***Efficiency and Reliability Concerns***: The proposed pipeline involves multiple modular components, including segmentation models, inpainting methods, and generative 3D models. However, the paper does not analyze the reliability of this multi-stage approach or present failure case studies. Furthermore, computational cost and runtime efficiency are not reported, which raises questions about the practical feasibility of the method in real-world or time-sensitive applications.\n\n***Methodological Clarity***:\n\n1. The regularization terms in the loss function (e.g., completeness, geometry, and physics terms) are under-explained. A more detailed formulation and intuition behind each term would improve clarity.\n\n2. The criteria for distinguishing background scene vs. foreground objects are not clearly defined. For example, are wall-mounted items considered part of the background?\n\n3. The Object Reconstruction (OR) metric is vaguely described. It remains unclear how the object/scene completeness is quantitatively evaluated—particularly in cases of occlusion or partial visibility.\n\nReference: \n[1] Holodeck: Language Guided Generation of 3D Embodied AI Environments \n[2] Digital Cousins: Automated Creation of Digital Cousins for Robust Policy Learning\n[3] Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation\n[4] Architect: Generating Vivid and Interactive 3D Scenes with Hierarchical 2D Inpainting\n[5] DeepPriorAssembly: Zero-Shot Scene Reconstruction from Single Images with Deep Prior Assembly", "was_revised": true, "final_justification": "Thank you to the authors for the detailed rebuttal and comprehensive failure case analysis, which addresses many of my concerns regarding the model’s reliability and robustness. The quantitative comparisons on geometry and physics failures are particularly helpful.\n\nMy remaining concerns primarily relate to:\n\n1. Computational efficiency – The reconstruction time of ~8 hours per scene may pose challenges for scaling to large datasets, especially when compared with generative methods such as Holodeck, which can produce physically plausible scenes within minutes. While I acknowledge the fidelity advantages of HoloScene, a more explicit discussion of this trade-off and potential paths for improving efficiency would strengthen the paper.\n\n2. Comprehensive baseline comparison – While the current comparisons cover several baselines, I recommend also including results against recent state-of-the-art reconstruction methods—both multi-stage approaches (e.g., CAST, DRAWER) and end-to-end methods (e.g., MIDI-3D)—evaluated on geometry and visual quality. This would more fully position HoloScene within the broader landscape of reconstruction approaches.\n\nI would like to increase my rating to 'borderline accept' considering all above-mentioned aspects.", "rebuttal": "We thank the reviewer for recognizing our unified framework and energy-based optimization design for simulation-ready environment reconstruction. \n\nWe now address the concerns.\n\n**Literature review:**  \nWe discuss all the papers mentioned in the review, and will include them in the final version.\n* **Digital Cousins [1]:**  \nIn the first row of Table 1, we include Digital Cousins (ACDC) [1], a single-image reconstruction-by-retrieval approach that operates under a fundamentally different setting than HoloScene. It retrieves plausible 3D assets based on similar semantics and layouts rather than precisely reconstructing the actual scene geometry and appearance. As such, it is not directly comparable to our method under the rendering or reconstruction metrics we report, since it does not aim to faithfully replicate the original scene. In contrast, HoloScene is designed to balance both simulation capability (as explored in Digital Cousins) and scene fidelity (as targeted by reconstruction methods).\n* **DeepPriorAssembly [2]:**  \nDeepPriorAssembly [2], like Digital Cousins [1], is a single-image reconstruction method and is not designed to faithfully reconstruct scenes consistent with image sets or video input—the primary focus of HoloScene. In Table 1 and the related work section, we already discuss methods with similar settings, including Gen3DSR and CAST. We will include DeepPriorAssembly [2] in the final version for completeness.\n* **Scenethesis [4]:**  \nScenethesis is a text-conditioned scene **generation** method, with a goal fundamentally different from ours. HoloScene focuses on reconstructing simulatable digital twins from images or videos. We also note that Scenethesis appeared on arXiv on May 5, 2025. According to the NeurIPS 2025 FAQ, *“Papers appearing online after March 1st, 2025 are generally considered concurrent to NeurIPS submissions. Authors are not expected to compare to those.”*\n* **Generation / retrieval-based scene generation:**  \nMethods such as Holodeck [3], Digital Cousins [1], Scenethesis [4], and Architect [5] generate 3D scenes from text or sparse inputs. While their outputs are also object-centric scene assets, their primary goal is to create diverse and plausible layouts—not to reconstruct high-fidelity digital twins faithful to input images in appearance or geometry. Consequently, these methods are not directly comparable to HoloScene under reconstruction or rendering metrics. In contrast, HoloScene integrates generative priors into a reconstruction pipeline that accurately replicates input videos and produces simulation-ready environments. It further leverages IsaacSim to ensure physical stability (see Eq. 5), as demonstrated in Fig. 4. \n\nWe will make these distinctions and contributions more explicit in the final version.\n\n**Baseline comparisons:**  \nThank you for the suggestions. We respectfully disagree with this concern. The methods mentioned in the reviews—Digital Cousins [1], Holodeck [3], Scenethesis [4], and Architect [5]—are retrieval or generation-based and not designed to produce 3D scenes that **align accurately** with input images or videos at *pixel-level appearance* and *centimeter-level geometry*, as elaborated in the previous response. These approaches target a different goal: generating diverse and plausible virtual scenes. While they reflect the state of the art in that domain, they are not representative of the current state of high-fidelity, simulation-ready reconstruction and twinning. As such, they are not directly comparable using our reconstruction or rendering metrics.\n\nIn contrast, our selected baselines are task-aligned and up-to-date. DP-Recon [6], included in our experiments, is a recent state-of-the-art method published at CVPR 2025 (June). ObjSDF++ [7], while introduced earlier, shares the same multi-view reconstruction setting as HoloScene and remains a strong and relevant baseline. Along with PhyRecon, these methods reflect the state of the field at the time of submission. We do not believe ObjSDF++ loses its relevance simply because it has been previously compared with DP-Recon. \n\nOverall, we believe our evaluation setup and baseline choice are fair, representative, and meaningful for assessing progress on this task.\n\n**Failure cases analysis:**  \nOur paper has shown failure cases caused by imperfections in foundation models, including inpainting from LaMa[8] and completion from Wonder3D[9], which can be found in Fig. 2, 3 in the main paper and Fig. 8 in the supplementary material and videos. We can witness that although Holoscene has achieved superior results over baselines, it still suffers from imperfect foundation model results. We will include a section on failure case analysis. Here is a brief analysis of failure cases:\n* **Inpainting:**  \nThe inpainting results with LaMa are not necessarily consistent with the background. (See the background inpainting color in the bottom of Fig. 2, and the sofa inpainting color in our object appearance results of Fig. 3) \n* **Completion:**  \nThe generation results of Wonder3D are not necessarily perfect as well. From Fig. 3, we can see the geometry of the back of the sofa has some artifacts (somewhat concave but not flat), which is due to the imperfect predictions from Wonder3D.\n* **Physics:**  \nFrom Fig. 8 in the supplementary material and the simulation videos in the supplementary material, in the experiments with the iGibson[10] dataset, we observe that some objects still deviate from original poses after applying gravity, which is due to imperfect predictions by LaMa and Wonder3D.\n* **Conclusion:**  \nThe failure cases are commonly from the foundation model's imperfect results. Holoscene could keep improving performance by leveraging more advanced foundation models.\n\n**Reliability against foundation model failures:**  \nHoloScene exhibits robustness to individual foundation model failures by jointly optimizing multiple energy terms and selecting the best among sampled solutions. Failures in one module typically introduce inconsistencies, which are reflected as higher energy, making such samples less likely to be selected. This contrasts with traditional multi-stage pipelines, where a single module failure can propagate downstream and compromise the entire output. This robustness is reflected in both our real-world examples—where foundation models are imperfect—and in benchmark metrics.\n\nThat said, this sampling-based robustness is not a guarantee against all types of failures. Catastrophic errors or insufficient sampling coverage, as shown in some of our failure cases, can still lead to imperfect final results.\n\n**Computation efficiency:**  \n*Simulation and rendering runtime*: Our simulation-ready environments support high-fidelity physical simulation and rendering at over 100 FPS on a single A6000 GPU. *Reconstruction runtime* The HoloScene reconstruction stage takes an average of 8 hours per scene. For comparison, DP-Recon [6], a state-of-the-art baseline, requires over 14 hours on average for similar reconstruction tasks.\n\n**Regularization terms:**  \nWe discuss completeness energy (L159-L167), geometry energy (L168-L175), and physics energy (L176-L183) in Sec 3.2. Because of the limit of maximum characters, we will include the intuition and a more detailed formulation for each energy term in the final version.\n\n**Criteria for foreground and background objects:**  \nWe define foreground objects as those that can be affected by **external forces**, such as gravity. For example, walls and wall-mounted structures (e.g., ceiling lights) are considered background objects, as they are fixed and unlikely to move under simple external forces. In contrast, objects merely leaning against walls—such as bookshelves—are considered foreground, as they can respond to physical interactions. \n\n**Object Reconstruction (OR) metric:**  \nThank you for pointing this out — there may have been a misunderstanding. The OR metric quantifies how many objects are present in the final reconstruction, regardless of their completeness. This metric is not designed to evaluate object completeness quantitatively; instead, the object-level CD, F1, and NC metrics in Table 2 serve that purpose. \n\nIn our baseline experiments, some small objects are missing from the reconstructions because no valid SDFs exist for these objects, preventing mesh extraction via marching cubes. This occurs due to their limited appearance in training images and the constraints of certain loss terms (such as the SDS loss in DP-RECON). In contrast, HoloScene addresses this issue by implementing balanced sampling across instances, ensuring robust reconstruction of all objects.\n\nExamples of missing objects in baseline reconstructions can be found in Figures 5-8 in the supplementary material. Figure 7 (the \"no physics\" column) shows DP-Recon reconstructing only one object on the shelf, while Figure 8 demonstrates that all baselines fail to reconstruct the small vase.\n\n\n[1] Dai, Tianyuan, et al. \"Automated creation of digital cousins for robust policy learning.\"\n\n[2] Zhou, Junsheng, et al. \"Zero-shot scene reconstruction from single images with deep prior assembly.\" \n\n[3] Yang, Yue, et al. \"Holodeck: Language guided generation of 3d embodied ai environments.\"\n\n[4] Ling, Lu, et al. \"Scenethesis: A language and vision agentic framework for 3d scene generation.\"\n\n[5] Wang, Yian, et al. \"Architect: Generating vivid and interactive 3d scenes with hierarchical 2d inpainting.\"\n\n[6] Ni, Junfeng, et al. \"Decompositional neural scene reconstruction with generative diffusion prior.\"\n\n[7] Wu, Qianyi, et al. \"Objectsdf++: Improved object-compositional neural implicit surfaces.\"\n\n[8] Suvorov, Roman, et al. \"Resolution-robust large mask inpainting with fourier convolutions.\"\n\n[9] Long, Xiaoxiao, et al. \"Wonder3d: Single image to 3d using cross-domain diffusion.\"\n\n[10] Li, Chengshu, et al. \"igibson 2.0: Object-centric simulation for robot learning of everyday household tasks.\""}, {"review_id": "P6jdcE3Btu", "reviewer": "Reviewer_Np4v", "summary": "This paper proposes HoloScene, a system for reconstructing physically plausible and interactive 3D scenes from monocular video. It models the scene as a structured scene graph with object-level geometry, appearance, and physics attributes. The method combines gradient-based and sampling-based optimization, and uses simulation feedback to ensure physical stability. The reconstructed scenes support editing, rendering, and deployment in interactive environments.", "questions": "1.You mention in the paper that your method follows the Gaussians-on-Mesh design from DRAWER. However, DRAWER is not included in the experimental comparisons. Given the similarity in task setting and overall pipeline, could you provide a detailed comparison with DRAWER, both quantitatively and qualitatively?\n\n2.Your method also shares many core components with CAST, including relation graph modeling, object-centric decomposition, and generative geometry completion. Could you clarify what you consider to be the key methodological differences and novel contributions beyond these prior works?\n\n3.More generally, please elaborate on the originality of your approach. What specific design choices or technical insights distinguish your method from existing systems like CAST and DRAWER, beyond the use of multiview input?\n\nIf you can address my concerns, I will increase my score.", "limitations": "yes", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 1, "strengths_and_weaknesses": "Strengths：\n\n1.The system is well-engineered and demonstrates solid integration of geometry, appearance, and physics.\n\n2.Results are visually compelling, and the method supports downstream applications such as interaction and simulation.\n\nWeaknesses：\n\n1.The method can be seen as a **multiview** extension of CAST[1], sharing similar core components such as relation graph modeling, object-centric geometry generation, structure-aware decomposition, and physics-based reasoning. I find the level of originality to be limited.\n\n2.Closely related works like DRAWER [2] and CAST are cited but not included in experiments, despite having **highly similar goals** and **setups**, making it difficult to assess whether the observed improvements come from the proposed method or from different input settings.\n\n3.The critique of CAST in the related work section is also questionable. For example, HoloScene highlights CAST’s reliance on generative models, but it similarly uses Wonder3D for multiview geometry synthesis.\n\nOverall, the contribution lies more in system integration and engineering effort than in novel methodology.\n\n[1]CAST: Component-Aligned 3D Scene Reconstruction from an RGB Image\n\n[2]DRAWER: Digital Reconstruction and Articulation With Environment Realism", "was_revised": true, "final_justification": "I appreciate the authors' rebuttal and acknowledge the engineering effort involved. However, I believe the proposed method remains largely an incremental extension of prior work like CAST, with limited methodological novelty. While technically sound, the contribution does not meet the originality bar expected for NeurIPS.", "rebuttal": "We thank the reviewer for appreciating HoloScene as a solid integrated system with visually compelling results and various downstream applications.\n\nWe now address the concerns.\n\n**Comparison with CAST [1]:**  \nHoloScene differs from CAST in problem setting, inference stage, and overall purpose. We provide a detailed comparison in Table 1 and the related work section, and summarize key differences below:\n\n* **Problem setting:**  \nCAST generates *component-aligned* 3D scenes from a single *image*, and its outputs do not necessarily match the input image in appearance or geometry (see CAST’s overview figure). In contrast, HoloScene reconstructs *simulation-ready digital twins* from a single *video*, aiming to replicate geometry, appearance, and physical plausibility from the input video.\n\n* **Inference stage:**  \nCAST relies on a generative model and uses differentiable optimization refining objects to avoid object collisions. HoloScene combines *reconstruction*, *generation* with *sampling-based optimization*, minimizing both observation energy (to match input video appearance) and regularization energy (for physical plausibility and object completion). While it leverages Wonder3D [3] for generation, reconstruction is guided by observed data.\n\n* **Purpose:**  \nCAST focuses on content creation with clean, component-aligned 3D assets from single images. HoloScene instead targets realistic, faithful, physically consistent reconstruction from video for simulation-ready environments. Both realism and fine-level fidelity are our key goals. \n\n**Comparison with DRAWER [2]:**  \nWe discussed DRAWER[2] in the related work section (L69-L71). Though HoloScene adapts Gaussians on Mesh from DRAWER[2], we have distinct differences with them. Here we will explain the detailed comparisons of HoloScene and DRAWER[2]. We also include a table below for the reviewer's reference.\n* **Focus:**  \nDRAWER[2] focuses on the *articulated* object reconstruction and lacks the shape completion and stable physics for some *rigid-body* objects. However, HoloScene focuses on *all* *rigid-body* objects in the input video, and reconstructs them with invisible region completion, collision avoidance, and physical stability in the simulation.\n* **Approach:**  \nIn DRAWER, *rigid-body* objects are decomposed following Video2Game[4]. From the paragraph of *Rigid object decomposition* in DRAWER's supp material page 9, we can see that, although *rigid-body* objects in DRAWER can be segmented from the SDF field using *3D bounding box*, DRAWER doesn't decompose every rigid-body objects and the objects lack the completion for invisible regions, so that they can not be guaranteed with physical stability in the simulation. Unlike DRAWER[2], HoloScene decomposes every *rigid-body* object in the video. Instead of direct segmentation from the SDF field, every object is modeled by its own SDF for convenient 3D decomposition. HoloScene also adapts a generative sampling approach to optimize the completion and stability of every rigid-body object. We show our quantitative comparison results with DRAWER in the table below. We compare on all the rigid-body objects in the scene.\n* **Experiments:**  \nWe compare the scene-level and object-level performance on the iGibson dataset with DRAWER[2]. We provide the ground-truth object 3D bounding boxes to DRAWER. We show the evaluation results in the table below. From the results, we can witness that DRAWER[2] fails to get complete geometry (according to object-level CD/F1/NC and PSNR/SSIM/LPIPS evaluation) and plausible physics from the input video, and can not ensure stability even for objects on the ground (according to the stability evaluation). \n\n|  Method   | Visual Input | Real-time Rendering | Twin Fidelity | Articulated Object 3D completion | Rigid-body Object 3D completion | Holistic Decomposition | Physics Capacity | Physics Optimization |\n|:---------:|:------------:|:-------------------:|:-------------:|:--------------------------------:|:-------------------------------:|:----------------------:|:----------------:|:--------------------:|\n|  DRAWER   |    video     |      &#x2713;       |   &#x2713;    |             &#x2713;             |            &#x2718;             |        &#x2718;        |     &#x2718;     |       &#x2718;       |\n| HoloScene |    video     |      &#x2713;       |   &#x2713;    |             &#x2718;             |            &#x2713;             |        &#x2713;        |      Scene       |   Diff & Sampling    |\n\n\n|  Method   | Level  | CD &darr; | F1 &uarr; | NC  &uarr; | PSNR &uarr; | SSIM  &uarr; | LPIPS &darr; | Stable (Ground) % &uarr; | Stable (All) % &uarr; |\n|:---------:|:------:|:---------:|:---------:|:----------:|:-----------:|:------------:|:------------:|:------------------------:|:---------------------:|\n|  DRAWER   | Scene  |   11.06   |   43.22   |   84.36    |    26.03    |    0.863     |    0.278     |           41.9           |         40.2          |\n| HoloScene | Scene  |   12.00   |   34.15   |   82.91    |    25.88    |    0.854     |    0.301     |           **74.4**           |         **71.1**          |\n|  DRAWER   | Object |   3.53    |   79.27   |   76.73    |    15.72    |    0.843     |    0.201     |           41.9           |         40.2          |\n| HoloScene | Object |   **3.17**    |   **81.31**   |   **78.13**    |    **16.55**    |    **0.863**     |    **0.185**     |           **74.4**           |         **71.1**          |\n\n**Technical contribution and system design novelty:**  \nWe agree that system integration is a key part of HoloScene’s technical contribution. However, the novelty and intellectual contribution go beyond integration. Our energy formulation and inference design are both novel, enabling HoloScene to jointly optimize for appearance-geometry alignment and physical-geometric plausibility. This combination makes HoloScene an effective and principled framework for simulation-ready scene reconstruction.\n\n\n[1] Yao, Kaixin, et al. \"Cast: Component-aligned 3d scene reconstruction from an rgb image.\" arXiv preprint arXiv:2502.12894 (2025).\n\n[2] Xia, Hongchi, et al. \"Drawer: Digital reconstruction and articulation with environment realism.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[3] Long, Xiaoxiao, et al. \"Wonder3d: Single image to 3d using cross-domain diffusion.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[4] Xia, Hongchi, et al. \"Video2game: Real-time interactive realistic and browser-compatible environment from a single video.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024."}, {"review_id": "uGjTjlKGHe", "reviewer": "Reviewer_BhAa", "summary": "This paper presents HoloScene, which aims to create an interactive, simulation-ready digital scene of an environment from a single input video. The main steps of the pipeline are to: (1) construct a scene graph hierarchy by recursively querying a VLM, (2) fit the 3d representation (parameterized by SDFs and 3D gaussians) via NeRF/3DGS-style optimization, (3) complete \"unseen\" properties (parts of objects not seen in input images + physics properties) by sampling generative priors (Wonder3D + LAMA) and a physics simulator (IsaacSim), (4) a final round of NeRF/3DGS-style finetuning to improve visual appearance. The resulting representation allows interactive 3D edition and the overlay of VFX effects (ie: set an asset on fire).\n\nThe authors compare HoloScene to prior work across indoor scenes from iGibson, Replica, and ScanNet++. Qualitatively, the physical interactions seem much more plausible and the scene geometry in unobserved regions looks a bit better. The supplement contains some nice illustrations of possible downstream applications, such as using the resulting environment in an interactive game.", "questions": "- Table 2: I understand that ObjSDF++ is not real-time, but we are still seeing a 2-4db PSNR gap that we don't typically see when comparing 3DGS to implicit representations. It's also really confusing how PSNR and SSIM/LPIPS don't really seem correlated, especially for the iGibson results where all of the PSNR values are very low. Could the authors provide a more detailed analysis of how to interpret these results? If accepted, I'd strongly recommend that the extra page in camera-ready version be used to provide some illustrative qualitative examples that illustrate these discrepancies.\n- Line 217-218: \"Thanks to its generative nature, resampling multiple times with different seeds yields diverse virtual observations even from the same viewpoints\" - I was confused about this statement - what would you want different observations from the same viewpoints? I would have assumed that multiview consistency is very important for 3D reconstruction, so I suspect that I am misunderstanding this statement.\n- Could the authors confirm that they commit to releasing the training code upon acceptance? This would greatly alleviate my reproducibility concerns and I'd likely adjust my score accordingly.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The visual results, although not 100% photorealistic at times, are impressive. I very much appreciate the videos the authors share in the supplement!\n- The overall design clearly requires quite a bit of thought, and the use of both gradient-based optimization + more discrete sampling to effectively use external priors seems like a sensible approach\n\nWeaknesses:\n- The pipeline involves a lot of discrete components (VLM, diffusion model, monocular depth prior, physics simulator, etc), and if any of the components behave incorrectly these errors might compound and the overall pipeline might degrade significantly. The authors note that the pipeline only works for indoor scenes at the moment but it would be interesting to get a sense of how the pipeline works in more challenging / less \"in-distribution\" scenes. Even if performance is not great, that would still help give us a sense of the approach's generalizability.\n- A major concern I have with this paper is that it omits a lot of training details that will make it very difficult to reproduce considering the complexity of the pipeline. A non-exhaustive list of missing important details include: loss hyperparameters, number of gradient optimization and sampling steps, average training time, what VLM/monocular depth models that are being used, and how the SDF penetration term is computed efficiently (a naive implementation seems prohibitively expensive). The authors mention additional details in the supplement, but I do not see any in the attached zip. In fairness, the authors committed to releasing code in the questionnaire, which would alleviate my concerns if they honor this commitment, but I would still like to see more details in the paper\n- The exposition of the writing could be improved - in particular I think that the authors should be more clearly state upfront what parts of the pipeline are novel contributions vs which have been inspired by prior work. For example, using a scene graph was originally proposed by Neural Scene Graphs (Ost el at, CVPR 2021), which should be explicitly cited and it's unclear if the use of a VLM to generate the graph is a novel contribution or something also proposed by other work.\n- The use of color highlighting in Table 2 is misleading / the results are hard to interpret (see questions section)\n\n\nTypos:\n- Physical plausible -> physically plausible", "was_revised": true, "final_justification": "Thank you for answering my questions - most of my concerns have been addressed and I will maintain my positive rating.", "rebuttal": "We thank the reviewer for appreciating our framework design and recognizing our videos as impressive results.\n\nWe now address the concerns.\n\n**Robustness of HoloScene:**  \nHoloScene exhibits robustness to individual foundation model failures by jointly optimizing multiple energy terms and selecting the best among sampled solutions. Failures in one module typically introduce inconsistencies, which are reflected as higher energy, making such samples less likely to be selected. This contrasts with traditional multi-stage pipelines, where a single module failure can propagate downstream and compromise the entire output. This robustness is reflected in both our real-world examples—where foundation models are imperfect—and in benchmark metrics.\n\nThat said, this sampling-based robustness is not a guarantee against all types of failures. Catastrophic errors or insufficient sampling coverage, as shown in some of our failure cases, can still lead to imperfect final results.\n\n**Outdoor scenes:**  \nHoloScene, like other compared methods, such as DP-Recon[1] and ObjSDF++[2], mainly focuses on and conducts experiments on indoor scenes. Outdoor scene setting features dynamic motions and unbounded geometry, which is out of our claimed scope. We will leave the outdoor simulation-ready environment reconstruction for future work.\n\n**Training details:**  \nWe will open-source our full codebase upon acceptance to support reproducibility. For the reviewer's reference, we provide additional training details in the table below and will include these parameters in the final version.\n\n*Gradient-based optimization (stage 1):*\n| Parameter | $\\lambda_{rgb}$ ($\\mathrm{E}_{obs}$) | $\\lambda_{mask}$ ($\\mathrm{E}_{obs}$) | $\\lambda_{depth}$ ($\\mathrm{E}_{obs}$) | $\\lambda_{normal}$ ($\\mathrm{E}_{obs}$) | total_steps |\n|:---------:|:------------------------------------:|:-------------------------------------:|:--------------------------------------:|:---------------------------------------:|:-----------:|\n|   Value   |                 1.0                  |                  0.5                  |                  0.5                   |                   0.1                   |    100k     |\n\n*Sampling-based optimization (stage 2):*\n| Parameter | $\\lambda_{rgb}$ ($\\mathrm{E}_{comp}$) | $\\lambda_{mask}$ ($\\mathrm{E}_{comp}$) | $\\lambda_{depth}$ ($\\mathrm{E}_{comp}$) | $\\lambda_{normal}$ ($\\mathrm{E}_{comp}$) | $\\lambda_{pene}$ ($\\mathrm{E}_{pene}$) | sampling # per instance |\n|:---------:|:-------------------------------------:|:--------------------------------------:|:---------------------------------------:|:----------------------------------------:|:--------------------------------------:|:-----------------------:|\n|   Value   |                  2.0                  |                  0.5                   |                  10.0                   |                   10.0                   |                  5.0                   |            3            |\n\n*Gradient-based Refinement (stage 3):* (for Gaussian refinements)\n\n\n| Parameter | $\\lambda_{l1}$ | $\\lambda_{ssim}$ |\n|:---------:|:--------------:|:----------------:|\n|   Value   |      0.95      |       0.05       |\n\n*Average Training time (one A6000 GPU):*\n\n| Stage |  0  |  1  |  2  |  3  |\n|:-----:|:---:|:---:|:---:|:---:|\n| Time  | 10m | 4h  | 4h  | 20m |\n\n*Models used:*\n\n\n| Model Type |  VLM   | monocular cues |\n|:----------:|:------:|:--------------:|\n| Model Name | GPT-4o |    Marigold[3]    |\n\n**SDF penetration term details:**  \nIn each iteration, we sample 4096 points uniformly in the 3D space, and calculate the SDF values of all instances at those points. We then apply $E_{\\mathrm{pene\\_sdf}} = \\sum_{\\mathbf{x} \\in R_i}\\sum_{k\\neq i}\\max\\bigl(0,\\,-g_k(\\mathbf{x}) - g_{i}(\\mathbf{x})\\bigr)$ on these points as the penetration loss to optimize. By uniformly sampling points, we can reduce the calculation complexity.\n\n**Scene graph inference, writing, and novelty:**  \nThank you for the suggestion. We will expand the related work section to include a discussion of prior work on scene graphs, including Neural Scene Graphs [Ost et al., CVPR 2021], and cite relevant references. \n\nWe do not claim the use of a VLM for scene graph inference as our main contribution. Rather, we emphasize that the **scene graph plays a critical role** in HoloScene for enforcing physically plausible interactions (see Tab. 3, L280–L286). While the idea of a scene graph itself is not novel, our **energy-based formulation and sampling-based inference framework built on top of the scene graph** is a core contribution of HoloScene. This integrated approach enables robust and physically consistent scene reconstruction from video.\n\n**Evaluation results analysis:**  \nHere we explain our evaluation results in Tab. 2. We will include a more detailed explanation in the final version.\n\n- *Real-time PSNR gap:* We adopt the Gaussian on Mesh (GoM) from DRAWER[4]. Compared with standard 3DGS, the position, orientation, scale, and other properties of every Gaussian are regularized, so they can align better with mesh surfaces. While this strategy limits the rendering quality, the real-time Gaussian rendering aligns with the accurate geometry, improving visual quality in downstream simulation applications. This is a key trade-off for attaining a simulation-ready environment. It is noticeable that the gap happens in DRAWER[4], GaMeS[5], and other GoM papers as well.\n\n- *Correlation of PSNR/SSIM/LPIPS:* PSNR and SSIM/LPIPS are correlated when we compare the real-time rendering quality in Tab. 2. But those three metrics sometimes are not necessarily correlated. For example, we can witness uncorrelated PSNR and SSIM/LPIPS in Table 1 of the Mip-NeRF 360[6] paper. \n\n- *iGibson results:* The object-level evaluation on iGibson scenes shows low PSNR values because the evaluation is performed on the whole object, including unseen regions, making it especially challenging. (L274-L275) Therefore, the PSNR/SSIM/LPIPS are lower than common reconstruction task.  We will add more qualitative examples in the camera-ready version.\n\n**Diverse results of resampling:**  \nThank you for pointing this out — there may have been a misunderstanding. By \"diverse virtual observations,\" we refer to the diversity in *invisible* regions when generative resampling with different seeds. This diversity helps us explore multiple plausible completions and select the one that best minimizes our regularization terms. \n\nImportantly, multi-view consistency is preserved through the use of an explicit 3D representation (GoM), ensuring that renderings from different viewpoints remain consistent with each other.\n\n**Code open source:**  \nWe will open-source all the code, data, and running scripts upon acceptance, and we believe this will encourage future research from the community.\n\n[1] Ni, Junfeng, et al. \"Decompositional neural scene reconstruction with generative diffusion prior.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[2] Wu, Qianyi, et al. \"Objectsdf++: Improved object-compositional neural implicit surfaces.\" Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.\n\n[3] Ke, Bingxin, et al. \"Repurposing diffusion-based image generators for monocular depth estimation.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2024.\n\n[4] Xia, Hongchi, et al. \"Drawer: Digital reconstruction and articulation with environment realism.\" Proceedings of the Computer Vision and Pattern Recognition Conference. 2025.\n\n[5] Waczyńska, Joanna, et al. \"Games: Mesh-based adapting and modification of gaussian splatting.\" arXiv preprint arXiv:2402.01459 (2024).\n\n[6] Barron, Jonathan T., et al. \"Mip-nerf 360: Unbounded anti-aliased neural radiance fields.\" Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2022."}]

Author rebuttal:

Meta-review: The paper firstly received mixed ratings, two positive and two negative. After rebuttal, one negative raise his/her score to be positive. Finally, only one reviewer remain negative, where the primary concern is the limited differences between the proposed method and existing method - CAST. 

The AC took very carefully reading on all comments and the submission. Although the AC agrees with that the paper is intrinsically a multi-view version of CAST, such an extension is not a trival task. In the literature, many works focused on 3D reconstruction from a single image, and also many other works focus on reconstrucion from multi-view images or a monocular video. The different problem settings usually causes different algorithm design. In the rebuttal, the author also provided many differences side by side. 

Thus, the AC recommends acceptance.
Decision: Accept (poster)