LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Control and Rendering
Authors: Delin Qu
Qizhi Chen
Pingrui Zhang
Xianqiang Gao
Bin Zhao
Zhigang Wang
Dong Wang
Xuelong Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Interactive Scene Reconstruction; Controllable NeRF; Language Embedding; Dataset;
Primary area: machine_vision
TL;DR: the first scene-level language-embedded interactive radiance field; interaction datasets;

Abstract: This paper scales object-level reconstruction to complex scenes, advancing interactive scene reconstruction. We introduce two datasets, OmniSim and InterReal, featuring 28 scenes with multiple interactive objects. To tackle the challenge of inaccurate interactive motion recovery in complex scenes, we propose LiveScene, a scene-level language-embedded interactive radiance field that efficiently reconstructs and controls multiple objects. By decomposing the interactive scene into local deformable fields, LiveScene enables separate reconstruction of individual object motions, reducing memory consumption. Additionally, our interaction-aware language embedding localizes individual interactive objects, allowing for arbitrary control using natural language. Our approach demonstrates significant superiority in novel view synthesis, interactive scene control, and language grounding performance through extensive experiments. Project page: https://livescenes.github.io.

LiveScene: Language Embedding Interactive Radiance Fields for Physical Scene Rendering and Control
Delin Qu
Qizhi Chen 3.2+
Pingrui Zhang 1.2
Xianqiang Gao?
Bin Zhao?
Zhigang Wang?
Dong Wang*1 21
Xuelong Li
Fudan University *Shanghai AI Laboratory *Zhejiang University
Physical Scene Modeling Capability
Parameters by Interactive Object Number
Readering Quality (PANT dB)
Dedomaaic 10GS
Cots
CoGS
CoNeRF
Model Para mereerseesrrrr (M)
MK. Plnee
HaperNoRF
CavannNG
K-Panes
NiRF
38 Stati
4d Defernable (344) Intenctive
Scene-level Language-based Control
Figure 1: LiveScene enables scene-level reconstruction and control with language grounding, Left: Language-interact articulated object control in Nerfstudio. Right: LiveScene achieves SOTA rendering quality on OmniSim dataset and exhibits significant advantage in parameter efficiency.
Abstract
This paper scales object-level reconstruction to complex scenes, advancing inter- active scene reconstruction. We introduce two datasets, OmniSim and InterReal, featuring 28 scenes with multiple interactive objects. To tackle the challenge of inaccurate interactive motion recovery in complex scenes, we propose LiveScene, scene-level minguage-ebeedee interactive radiance field that efficiently recon- structs and controls multiple objects. By decomposing the interactive scene into local deformable fields, LiveScene enables separate reconstruction of individual ob- ject motions, reducing memory consumption. Additionally, our interaction-aware language embedding localizes individual interactive objects, allowing for arbitrary control using natural language. Our approach demonstrates significant superiority in novel view synthesis, interactive scene control, and language grounding perfor- mance through extensive experiments. Project page: https//livescenes.ittuus github.io
Introduction
Interactive objects are prevalent in our daily lives, and modeling interactable scenes from the real physical world plays an essential role in various research fields, including content genera- virtual reality [50] 9 32). robotics [1] 44), and world understanding [16] (33. This paper tackles the challenging and rarely explored task of reconstruct- ing and controlling multiple interactive objects in complex scenes from single, casually captured monocular video without previous independent modeling of geometry and kinematies. Prior research on interactable scene modeling, such as CoNeRF (20 and K-Planes [10), typically adopts joint
tion 33 35 52), animation 588 37 30
Authors contributed equally: (dlqp22 wwt.m/00.m..ee.ommmmmmmmmmmmmmma www.ffudadduuuc.mm edu.enn
Corresponding author: dongwangg www.wnng.dww36. www.:e/wwwww@lgm.mlcoooo .om.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

modeling approach, combining spatial coordinates and all interaction variables as input and repre- senting interactive scene by either implicit MLPs or feature planes. Meanwhile, CoGS 63 learns parameter offsets for different scene parts using multiple independent MIPs after establishing a 3ID
deformable Gaussian scene. However, these methods primarily focus on capturing interactions for a
single object within a clear background, such as single drawer, toy car, or face [66] 20) 63 688. As
modeling extends from single objects to multiple objects in complex scenes, as shown in Figure. | the interaction spaces become increasingly high-dimensional, complicating these methods for accurate modeling and significantly increasing computational time and memory cost, e.g., 4x A GPU for 2 weeks to converge training in CoNeRF 20 and 500M Gaussian storage for regular indoor living room in CoGS [63]. Moreover, natural language is an intuitive and necessary interface for interacting with 3D scenes, but language embedding of interactive scenes faces an even more daunting challenge: interaction variation inconsistency. For instance, methods like LERF 23 and OpenNeRF [67], which distill CLIP features into static 3D fields, suffer from significant failures when confronted with scene topology structure changes induced by interactions, such as the distinct structures variation of a cabinet before and after opening.
To address these challenges, we propose LiveScene, the first scene-level language-embedded radi- ance fields, which compresses high-dimensional interaction spaces into compact 4D feature planes, reducing model parameters while improving optimization effectiveness. LiveScene models multi- ple object interactions via novel high- nigh-dimensional factorization, decomposing the scene into local deformable fields that model individual objects with multi-scale 4D deformable feature planes. To achieve independent control, we introduce a multi-scale interaction probability sampling strategy for factorized local deformable fields. Our interaction-awan language embedding method generates varying language embeddings to localize and control objects under arbitrary states, enabling natural language control. Finally, we construct the first scene-level physical interaction datasets, OmniSim and InterReal, featuring 28 scenes with 70 interactive objects for evaluation.
Experiment results show that our approach achieves SOTA novel view synthesis quality, outperform- ing existing best methods by +9.89, +1.30, and +1.99 in PSNR on the CoNeRF Synthetic, OmniSim #chanllenging, and InterReal #chanllenging subsets, respectively. Surpassing LeRF [23], LiveScene significantly improves language grounding accuracy by +65.12 of mIOU on the OmniSim dataset. Notably, our method maintains a lightweight, constant model parameter of 39M, scaling well with increasing scene complexity, shown in Figure. [ Contributions can be summarized
We propose LiveScene, the first scene-level language-embedddd interactive radiance field, which efficiently reconstructs and controls complex physical scenes, enabling manipulation of multiple articulated objects and language-based interaction.
We propose factorization technique that decomposes interactive scenes into local deformable fields and samples relevant 3D points, enabling control of individual objects. Additionally, we introduce an interaction-avee
language embedding method that generates varying embeddings, allowing for language- based control and localization.
We construct the first scene-level physical interaction dataset OmniSim and InterReal, containing 28 subsets and 70 interactive objects for evaluation. Extensive experiments demonstrate our SOTA performance and robust interaction capabilities.
2
Related Work
Dynamic Scene Representation. Extending NeRF [38 to dynamic scene reconstruction has made significant progress. Related methods can generally be categorized into time-varying methods, deformable-canonicn methods, and hybrid representation methods. The time-varying methods [ 54 655 typically model the radiance field directly over time, but struggle to separate dynamic and static objects. Deformable-canon field and static canonical space, modeling 4D by warping points with deformable field to query the canonical features. However, these methods face challenges in scene topology changes [10]. Hybrid representation methods, on the other hand, have achieved high-quality reconstruction and fast rendering by utilizing time-space feature planes 47
methods [1] 31
42 60] decouple dynamic deformable
10
3 4D hash encoding (56). dynamic
voxels [57], or triple fields [4]]. Recently, several works 36
62
59] 27 have introduced 3D
gaussians 21 into dynamic scene reconstruction, achieving high-quality real-time rendering speeds.

---

Comput
Friey and Sille
Aple
danthatte
ivoliacs
Capaarn
Incation
letera Saa-oaar larguge plans
Bradering Coeard
Figure 2: The overview of LiveScene. Given camera view and control variable of one specific interactive object, a series of 3D points are sampled in a local deformable field that models the interactive motions of this specific interactive object, and then the interactive object with novel interactive motion state is generated via volume-rendering. Moreover, an interaction-awer language embedding utilized to localize and control individual interactive objects using natural language.
However, these methods limited reconstructing dynamic scenes and lack the ability control and understand interactive scenes.
3D Vision-language Fields. Vision-language foundational models
with strong general- izability and adaptability inspires numerous language embedded scene representation for 3D scene understanding
45
25
70, such open-vocabulan segmentation 34
35
24, 3D visual question LeRF [23 is the first to
answering [S
6
19
18 and 3D language grounding [17] 46
5
18
achieve open-vocabulle 3D queries by combining CL.IP 45 and DINO
4 with NeRF through
paradigm and leverages hierarchical embeddings to address 2D knowledge-distiling issues from SAM. LEGaussians [48] and LangSplat [3 integrate semantic features into 3D gaussians [2]] and achieve precision language query and efficient rendering. However, these methods are limited to static scene understanding and fail to generalize when the interactive scene topology changes.
feature distillation. Open-NeRF [67] introduces an
Controllable Scene Representation. Manipulating reconstructed assets or neural fields is of significant importance for avatar and robotic tasks [8
12
20 23
CoNeRF [20 pioneered this
effort by extending HyperNeRF [2] and introduce fine-graine controlable neural field with 2D attribute mask and value annotations. CoNFies [64] proposes an automatic controllable avatar system and accelerates rendering by distilling. More recently, CoGS [63] leveraged 3D Gaussians [21] to achieve real-time control of dynamic scenes without requiring explicit control signals. However, these methods typically lack natural language interaction capabilities, relying solely on manual control. Furthermore, most works focus on single few object interactions, disregarding the interaction between different scene parts, limiting their real-world applications.
3 Methodology
We aim to establish a representation that models a interactive articulated objects in a complex scene from a monocular video via a rendering-
self-supervised manner. Control variables K == [Kg, N2, .., Ka] indicating object motion states and camera poses each video frame are given. The overview of LiveScene is shown in Figure. Section. 3.1 introduces the high-dimenssonal interactive space modeling and challenges. Section. 3.2 presents a multi-scale interactive space factorization and sampling strategy to compress the high- dithnniienii interactive space into local 4D deformable fields, and model complicated interactive motions individual objects. Section. 3.3 introduces interaction-aware language embedding method localize and control interactive objects using natural language.
3.1 Interactive Space
Assuming non-rigidly interactive with control variables x [Kr, K2,..., .... Ka. corresponding to a objects, we delincate its representation by a high-dimensional function:
y p(x, K. H);),
(I)
where / is the model of the representation, x E 3 are spatial coordinates, K. R are control variables, is set of optional additional parameters ge.g., the view direction), and 0 stores scene information. The function outputs scene properties y for the given position and sampling
3

---

from a ray where y can be represented with color, occupancy, signed distance, density, and BRDF parameters. This paper focuses on color, probability, and language embedding. Distinguishing from 3d static scene or 4d dynamic scene modeling, the sampling point p [x|<] R3+0 in the
interactive scene is high-dimensional and variable in topological structure, complicating the scene feature storage and the optimization of representation model, leading to significant time-consuming
or memory-intensive training in [20 63
3.2 Multi-scale Interaction Space Factorization
For the (3 + a) d-dmensiional interactive space containing a control variables, we aim to explicitly represent the high-dimensional space in a concise and compact storage, thereby reducing memory usage and improving optimization. As illustrated in Figure, 2 and Figure. TO] objects exhibit mutual independence, and interaction features are distributed in the (3 + a ----mmmndmional interactive space and aggregate into cluster centers. Thus, there exists set of hyperplanes that partition the space into disjoint regions, with each region containing local 4D deformable field, as shown in Figure. 3 Hence, the interaction features at sampling point p E R can be projected into compact 4-dimensional space R in Figure. 3 by transformation.
Multi-scale Interactive Ray Sampling. We consider using ray sampling per- form the projection transformation. As shown in Figure. assuming a ray r(t) Tox|ox) + +.t d dæ.] with origin Ox E R.3 Ox E Ra, and direction dx E R3, dd E RR, the ray intersects with the interaction region at a point p = [x|<] E Rsta where x E R is the 3D position and к E Rº is the interaction variables. For a given intersection point p. the de- formable features can be retrieved from the corresponding local 4D deformable field by maximizing sampling probability P:
projociall samplex P. [xls[]]]
my samples
|x x] g(
Planet
R
R2
R
P
a) Local deformable fields
b) Compact multiscale interaction
fenture storage
Figure 3:
Illustration of hyperplanar factorization for compact storage. We maintain multiple local de- formable fields for each interactive object region R,, and project high-dimensional interaction features into a compact 4D space, which can be further compressed into multiscale feature planes.
Pu. [x)x())),
arg max P P= (
where Pa, and 0 are the 4D sampling point and probability features at position x from 3D feature planes. The maximum argument operation of probability decoder ( 0) maps the interaction variables & to the most probable cluster region in the 4D space, and can be optimized by minimizing the focal loss Liccus of mask across all the training camera views:
Crocus B (1- M, 1og()P))) M.
Σ M, M.
where M is the ground truth mask label, is the probability map rendering from the interactive probability field, B is the balancing factor, and "y is the focusing parameter.
Next, the deformable features are used render local deformable scene color at the sampling point p. In this way, the high-dimensional interaction features are factorized into 4D space by a lightweight transformation modeling with interaction probability decoder and 3D feature planes in Figure. supervised by deformable masks M. Moreover, leveraging K-Planes [0). the multiple 4D local deformable space can be further compressed in only C 6 feature planes. We iteratively sample from coarse to fine within the multi-scale feature plane, retrieving the maximum probability interaction variables Ky and indices u at each scale.
Feature Repulsion and Probability Rejection. A latent challenge is that optimizing the interaction probability decoder with varying masks can lead to blurred boundaries in the local deformable field, further causing ray sampling and feature storage conflicts. As illustrated in Figure. a), consider two adjacent local deformable regions R. and R,j, and a point p in high-dimensional space, suppose moves from the cluster center of RR towards the cluster center of R2, then the probability of belonging to R. gradually decreases, while the probability of p belonging to R, increases. To avoid sampling conflicts and feature oscillations at the boundarics, introduce repulsion loss for ray

---

pairs ry), and amplify the feature differences between distinct deformable regions, promoting the separation of deformable field:
Crepuls = ELU(K- - -I(M, O (F) - -f))))
(4)
where K a constant hyperparameter, M, and M, are the ground truth mask of rays. F, and J3 are the last-layer features of interaction probability decoder in Figure. 2 During training, randomly select ray pairs and apply Crepuls to enforce the separation of interactive probability features across local deformable spaces. The initiative is inspired by [24], which demonstrated the efficacy of repulsive forces in disambiguating 3D segmentation results.
Additionally, the probability rejection is proposed to truncate the low-probability samples if the maxi- mum deformable probability P at sample p is smaller than threshold 8, and selects the background feature directly. The operation is defined as:
arg max, (P:), if P,28
(5)
otherwise
As shown in Figure, 4b), the proposed operations help the model achieve higher rendering quality, demonstrating their effectiveness in alleviating boundary sampling conflicts.
H
FRt sam
Iision
hygier plane
probability rejection
Be
wio repulsion and rejection
Ours
b) Rendering Quality Comparison Figure 4: Illustration of a) boundary sampling conflicts, b) rendering quality comparison.
a) Local Feature Conflicts
3.3 Interaction-Aware Language Embedding
Language embedding in interactive scenes is complex and storage-intensive, as 3D distillation faces the dual challenges high-dimensional optimization and interaction scene variation inconsistency, such as the distinct topological structures of transformer toy before and after transformation, leading the failure of SAM (25 segmentation or LERF (23) grounding. As shown in Figure. E leveraging the proposed multi-scale interaction space factorization of 3.2] we efficiently store language features in lightweight planes by indexing them according to maximum probability sampling instead of 3D fields in LERF. For any sampling point p. project it onto p.. [x]x[u)], retrieving local language feature group by index d, and perform bilinear interpolation using K., to obtain a language embedding that adapts interactive variable changes from surrounding clip features. By interpolating language embeddings, our method not only perceives topological structure changes but also achieves storage complexity of OCC' x x dim), much smaller than language distillation methods like LERF that operate in 3D scenes, where dim is the dimension of CLIP feature.
Dataset
To our knowledge, existing view synthetic datasets for interactive scene rendering are primarily limited to a few interactive objects [66] 20 63 68) due to necessitating a substantial amount of manual annotation of object masks and states, making it impractical to scale up to real scenarios involving multi-object interactions. To bridge this gap, we construct two scene-level, high-quality an- notated datasets to advance research progress in reconstructing and understanding interactive scenes: OmniSim and InterReal, as shown in Figure. 5 Besides, we use the CoNeRF Synthetic and Control- lable 20 dataset for evaluation as well. 1) OmniSim Dataset rendered through OmniGibson (2) simulator, leveraging indoor scene models: #rs, # #ihlen, #beechwod, #merom, #poma ria, #wa nscott and #benevo encee By varying the rotation vectors of the articulated objects' joints and the camera's 's trajectory within the scene, we generated 20 high-definition subsets, each
5

---

OmniSim Behavior Synthetic and Inter Real Dataset
camera trajectory,
cumples tramiation
#28 Interactive Subsets with Millions Sample including RGB, Depth, Segmentation, Camera Poses, Interaction Variables, Object Captions Modalities
Figure 5: Overview of the OmniSim and InterReal datasets.
consisting of RGBD images, camera trajectory, interactive object masks, and corresponding object state quantities at each time step. InterReal Dataset captured from 8 real Interactable scenes and finely annotated with interaction variables and masks, camera poses encompassing multiple objects, and articulated motion variables. More details be found in the supplementary.
Experiment
Baselines. We compare LiveScene with the existing 3D static rendering methods [38 39
4D deformable methods [TO] 42 41
and controllable scene reconstruction methods (2 63 Note
63 based on Deformable Gaussian (62] since the official code is unavailable. Additionally, we extended K-Planes (10) from C2 planes to Cl planes, denoted as MK-Planes, where a represents the number of interactable objects in dynamic scenes. By leveraging the fact that each instance occupies distinct region, we further compressed the model, denoted as MK-Planes", requiring only 3 + 3a planes.
that we reimplemented CoGS
Implementation. LiveScene is implemented Nerfstudio [51] from scratch. We represent the field as multi-scale feature plane with resolutions of 512 x 256 x 128, and feature dimension of 32. The proposal network adopts a coarse-to-fine sampling process, where each sampling step concatenates the position feature and the state quantity the query for the 4D deformation probability field, which a I-layer MLP with 64 neurons and ReLU activation. We use the Adam optimizer with initial learning rates of 0.01 and cosine decay scheduler with 512 warmup steps for all networks. The model is trained with an NVIDIA A100 GPU for 80k steps on the OmniSim dataset and 100k steps on the InterReal dataset, using batch size of 4096 rays with 64 samples. More implementation can be found in the supplemental materials.
5.1 View Synthesis Quality Comparison
Evaluation on CoNeRF Synthetic and Controllable Datasets. We report the quantitative results on CoNeRF Synthetic and Controllable scenes in Table. LiveScene outperforms all the existing PSNR, SSIM, and LPIPS metries methods on CoNeRF Synthetic scenes with large margin. particular, LiveScene achieves 43.349, 0.986, and 0.011 1 in PSNR, SSIM, and LPIPS, respectively, outperforming the second-best method by 9.894, 0.009, and 0.053. On CoNeRF Controllable, LiveScene achieves the best PSNR of 32.782 and comparable SSIM and LPIPS the SOTA methods. According the novel view synthesis results in Figure. 6 LiveScene achieves more detailed and higher rendering quality, demonstrating the effectiveness of LiveScene in modeling object-level interactive scenarios.
Evaluation on OmniSim Datasets. OmniSim dataset categorized into interaction level subsets: #easy, #medium, and #challenging, based on the number of interactive objects in each scene. As shown in Table. 22] LiveScene achieves the best PSNR, SSIM, and LPIPS on interaction level subsets of OmniSim, with average PSNR, SSIM, and LPIPS of 158, 0.962, and 0.074, respectively. Notably, substantial performance degradation is observed across methods as the quantity and complexity of interactive objects increase, c.g., CoGS 663 experiences 3.641 dB PSNR drop from

---

Table 1:
Quantitative results on CoNeRE synthetic and controllable datasets. LiveScene achieves
the best results in all metrics on synthetic scenes and the best PSNR on the controllable datasets.
CoNeRF Syntetic
CoNeRF Controllable
Method
PSNR1
SSIM1
LPIPS.
PSNR1
SSIM
LPIPS
NeRF 38
25 299
0.843
0.197
28.795
0.951
0.210
InstaniK
39
27.057
0,903
0.230
26.391
0.884
0.278
3DGS 21
32.576
0.977
0.077
25.945
0.834
0.414
NeRF Latent [38
28.447
0.939
0.115
32,653
0.981
0.182
Nerfies
32.274
0.981
0,180
HyperNeRRi l
25.963
0.854
0.158
$2.520
0.981
0.169
k K-Planes [00
33.301
0.933
0.150
31.811
0.912
0.262
Co 20
27. 27.868
0.898
0.155
32.061
0.979
0.167
CoNeR 20
32.394
0.972
0.139
32.342
0.981
0,168
CoGS [6]]
33.455
0.960
0.064
32,601
0.983
0.164
LiveStene (Ours)
43.349
0.986
0.011
32.782
0.932
0,186
Table 2: Quantitative results on OmniSim Dataset. LiveScene outperforms prior works on most
metrics and achieves the best PSNR on the #challenging subset with a significant margin.
#Fasy Sets
#Medium Sets
#Challenging Sets
#Avg (all 20 Sets)
Method
PSNR1 SSIM LPIPS.
PSNR
SSIM LPIPS..
PSNR SSIM LPIPS.
PSNR
SSIM LPIPS..
NcRF 38
25.817
0.906
0.167
25.645
0.928
0.138
26.364
0.927
0.128
25.776
0.916
0.153
InstantNGP 39
25.704
0.902
0.183
25.627
0.930
0.140
26.367
0.920
0.143
25.706
0.914
0.164
HyperNeRI
42
30.708
0.908
0.316
31.621
0.936
0.265
27.533
0.897
0.318
30.748
0.917
0.299
K K-Planes
10
32.841
0.952
0.093
32.548
0.954
0.100
29.833
0,937
0.118
32.573
0.952
0.097
CoNeRF 200
32.104
0.932
0.254
33.256
0.951
0.207
30.349
0.923
0.238
32,477
0.939
0.234
MK-Plancs*
31.630
0.948
0.098
31,880
0.951
0.104
26.565
0.887
0.218
31.477
0.946
0.106
MK-Plancs
31.677
0.948
0.098
32.165
0.952
0.099
29.254
0.933
0.119
31.751
0.949
0.099
CoGS [63]
32.315
0.961
0.108
32. 447
0.965
0.086
28.701
0.970
0.073
32.187
0.963
0.097
LiveScene (Ours)
33.221
0.962
0.072
33,262
0.965
0.072
31.645
0.948
0.093
33.158
0.962
0.074
Quantitative results on InterReal Dataset. Our method outperforms others in most settings, with significant advantage of PSNR, SSIM, and L.PISS on the #tchallenging subset.
Table 3:
#Medium Sets
#Challenging Sets
#Avg (all Sets)
Method
PSNR1 SSIM LPIPS.
PSNR1 SSIM
LPIPS
PSNR1 SSIM LPIPS.
NeRF 38
20,816
0.682
0.190
21.169
0.728
0.337
20.905
0.694
0.227
In
39)
21.700
0.776
0.215
21.643
0.745
0.338
21.686
0.769
0.245
HyperNeRF
42
25.283
0.671
0.467
25.261
0.713
0.517
25.277
0.682
0.480
K-Planes [10
27.999
0.813
0.177
26.427
0.756
0.331
27.606
0.799
0.215
CoNeRF 20
27.501
0.745
0.367
26.447
0.734
0.472
27.237
0.742
0.393
CoGS [63
30.774
0.913
0.100
X
X
30.774
0.913
0.100
LiveScene (Ours)
30.815
0.911
0.066
28.436
0.846
0.185
30.220
0.895
0.096
#easy to #challenging. While LiveScene maintains a relatively stable high performance across all subsets, demonstrating its robustness in modeling complex interactive scenarios.
Evaluation on InterReal Datasets. We divided InterReal dataset into #medium and #challenging subsets. In Table. 33 CoGS [63] underperforms compared to LiveScene on the #medium subset and fails to converge when faced with long camera trajectories and a large number of interactive objects in the scene (#challenging), highlighting the limitation of existing methods in modeling real-world interactive scenarios. contrast, LiveScene achieves the highest PSNR of 28.436 and the lowest LPIPS of 0. 1858 on the #challenging subset, indicating its superiority in modeling real-world large-scale interactive scenarios.
View Synthesis Visulization. Figure. presents the novel view synthesis results of LiveScene and the SOTA methods on OmniSim dataset. The results reveal that LiveScene generates more detailed results than SOTA methods, particularly in complex interactive scenarios. For instance, on the #pomaria scene featuring an openable dishwasher, CoNeRF 20) fails to capture details, while CoGS (33) and MK- exhibit residual artifacts. In contrast, our method accurately reconstructs the internal details. Another challenge arises in the #rs scene, where other methods struggle to reconstruct distant and static objects. In comparison, our method not only overcomes the challenging problem of dramatic topology changes in interactive scenes but also maintains the ability to reconstruct high-quality static scenes.
7

---

GT
HyperNeRF
CoNeRF
CoGS
Ours
Figure 6: View Synthesis Visualization on CoNeRF Controllable Dataset. The proposed method
achieves higher-quality rendering results compared with the existing methods.
GT
CoNeRF MKPlanes
CoGS
Ours
GT
CoNeRF
MKPlanes"
CoGS
Ours
#11
#ihlen
#pomaria
#scene
#rs
#ihlen
#pomaria
Figure View Synthesis Visualization on OmniSim Dataset. We compare our method with SOTA methods on RGB rendering across three scenes: #rs, #ihlen, and #pomaria. Boxes of different colors represent distinct interactive objects within the scene.
5.2 Language Grounding Comparison
We assess language grounding performance on the OmniSim dataset using mIOU metric. Figure suggests that our method obtains the highest mIOU score, with an average of 86.86. In contrast, traditional methods like LERF (23) encounter difficulties locating objects precisely, with an average mIOU of 21.74. Meanwhile, 2D methods like SAM [25) fail to accurately segment the whole target

---

mIOU f
setting
SAM 25
LERF 23
Ours
Minoo
OmniSin
#easy
61.58
23.60
86.94
#medium
55.13
19.40
86.32
#challenging
63.86
19.87
90.41
FTT SAM
LIRR
Obrra
#avg
59.11
21.74
86.86
InteRRoll
#medium
93.27
27.63
84.37
#challenging
91.50
34.39
91.90
#avg
92.82
29.32
86.26
SAM
Figure 8: Language Grounding Performance OmniSim Dataset left): Our method gains the highest mIOU score. right): LiveScene's grounding exhibits clearer boundaries than other methods.
under specific viewing angles, as objects appear discontinuous in the image. Conversely, method perceives the completeness of the object and has clear knowledge of its boundaries, demonstrating its advantage in language grounding tasks.
#1 w/o multiscale factor
LiveScene
GT
#l wio mulliscale factor
LiveScene
interaction-cllevat
interaction-relevand
Figure 9: Rendering and Grounding Performance for #1 and #6. above): Multi-scale factorization greatly boosts the performance RGB rendering and geometry reconstruction. below): Without view consistency, the model struggles when objects have similar appearances.
Internctive
step
slep
Figure 10: Learning process of the probability fields from 0 to 1000 training steps. The model progressively converges to the vicinity of the interactive objects, establishing interactive regions.
5.3 Ablation Study
In this section, we present ablative studies to investigate the effectiveness of each component in LiveScene. We selected scene from the #medium subset, 2 scenes from #easy subset of OmniSim dataset, and 1 scene each from the #medium and #challenging settings for InterReal. Notably, ground-truth state quantities are only available in OmniSim, not in InterReal. Therefore, we GT quantities OmniSim and introduce learnable variable on InterReal to infer state changes. Figure. reports the rendering quality and grounding performance for #1 and #6.
Effectiveness of components. As illustrated in Table. 4 the multi-scale factorization significantly improves the rendering performance on both datasets, with PSNR on OmniSim increasing from 31.74 11.74 to 35.094, shown in #1. Introducing learnable variables for each frame (#2) yields corresponding improvements on InterReal dataset since this latent code can perceive the change in object states. The feature repulsion loss and probability rejection (#3 and #4) together make rendering quality better in InterReal as well as in OmniSim dataset. As for grounding, #5 shows that rendering embeddings along ray [26 69 struggles locate objects precisely. Ensuring view consistency further boosts grounding performance on OmniSim, as demonstrated in #6.
Probability field training. We provide additional experiments Figure. of the disjoint regions to illustrate the learning process of the probability field from 0 to 1000 training steps. The

---

results demonstrate a clear trend that, as training advances, the proposed method can progressively converge to the vicinity of the interactive objects, thereby establishing interactive regions. With the establishment of the probability field, the model can focus different interactive objects and guide the sampling process by maximizing probability, thereby achieving disentanglement of interactive scenes.
Table 4: Ablation Study on the subset of InterReal and OmniSim Datasets.
#settings (readering)
InterReal
OmnnSSi
#exp
SSIM
LPIPS_
SSIM:
LPIPS]
Depth L1J
31,740
0.118
0.238
0.819
0.226
35,094
0.969
0.059
0.086
29.377
0.865
0.162
29.959
0,883
0.131
34.989
0.969
0.059
0
30:123
0.884
0.132
34.977
0.967
0.061
0.086
LiveScene
30.591
0,896
0.115
35.254
0.971
0.057
0.042
Rocttings (grounding)
ml0U
mlOU
#5
30,40
32.87 71 64
#6 LiveScene
93.10
93.02
78.52
multi-scale factorization.
It learnable varialle
penhubility rejection, OmniSim.
maximum prolubility embeds
VI: Intenction-sware languge embhatding. denotes
Conclusion and Limitation
We present LiveScene, the first language-embeddedd interactive neural radiance field for complex scenes with multiple interactive objects. A parameter-efficien factorization technique is proposed decompose interactive spaces into local deformable fields to model individual interactive objects. Moreover, we introduce a novel interaction-awant language embedding mechanism that effectively localizes and controls interactive objects using natural language. Finally, We construct two challeng- ing datasets that contain multiple interactive objects in complex scenes and evaluate the effectiveness and robustness LiveScene.
The control ability of LiveScene is limited by label density. Additionally, our natural language control is currently restricted to closed vocabulary, and it is inherently tied to the capabilities of the underlying foundation model, such OpenCLIP. In future work, we plan extend our method to enable open- vocabulary grounding and control, increasing the model's flexibility and range of applications.
Limitations:
Acknowledgements. This work is partially supported by the Shanghai Al Laboratory, National Key R&D Program of China (2022ZD0160101), the National Natural Science Foundation of China (62376222), and Young Elite Scientists Sponsorship Program by CAST (2023QNRC001).
10

---

References
[1] Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian Reid, Stephen Gould, and Anton Van Den Hengel. Vision-and-languaal navigation: Interpreting visually-gre grouudy- navigation instructions in real environments. In Proceedings of the IEEE conference on computer vision and puttern recugnition, pages 3674-3683, 2018.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, al. On opportunities and risks of foundation models. arXiv preprint arXiv:2108.07255. 2021.
[3] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 130-141, 2023.
[4] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[5] Boyuan Chen, Fei Xia, Brian lchter, Kanishka Rao, Keerthana Gopalakrishnan, Michael S Ryoo, Austin Stone, and Daniel Kappler. Open-vocabulane queryable scene representations for real world planning. In IEEE International Conference on Robotics and Automation, pages 11509-11522. IEEE, 2023.
[6] Sijin Chen, Xin Chen, Chi Zhang, Mingsheng Li, Gang Yu, Hao Fei, Hongyuan Zhu, Jiayuan Fan, and Tao Chen. LI3da: Visual interactive instruction tuning for omni-3d understanding reasoning and planning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24428-26438,2 2024.
[7] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia, pages 1-9, 2022.
[8] Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, et al. Foundation models in robotics: Applications, challenges, and the future. arXiv preprint arXiv:2312.07843 2023.
[9] Carlos Flavián, Sergio Srääiez-Sáncher and Carlos Orús. The impact of virtual, augmented and mixed reality technologies on the customer experience. Journal of Business Research, 2019.
[10] Sara Krillvich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479 12488, 2023.
[11] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision. pages 5712 2021.
[12] Stephan Garbin, Marek Kowalski, Virginia Estellers, Stanislaw Szymanowicz. Shideh Rezaeifar, Jingjing Shen, Matthew A Johnson, and Julien Valentin. Voltemorph: Real-time, controllable and generalizable animation of volumetric representations. In Computer Graphics Forum, volume 43, page e15117. Wiley Online Library, 2024.
[13] Rahul Goel, Dhawal Sirikonda. Saurabh Saini, and PJ Narayanan. Interactive segmentation radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4201 1211-4221,200 2023.
[14] Zoey Guo, Yiwen Tang, Ray Zhang, Dong Wang, Zhigang Wang. Bin Zhao, and Xuelong Li. Viewrefer: Grasp the multi-view knowledge for 3d visual grounding. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13372-15383 2023.
[15] Yining Hong, Chunru Lin, Yilun Du, Zhenfang Chen, Joshua B Tenenbaum, and Chuang Gan. 3d concept learning and reasoning from multi-view images. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2023.
[16] Yining Hong. Zishuo Zheng, Peihao Chen, Yian Wang. Junyan Li, and Chuang Gan. Multiply: multisensory object centric embodied large language model in 3d world. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages (6406-26411 2024.
11

---

[17] Chenguang Huang. Oier Mees, Andy Zeng, and Wolfram Burgard. Visual language maps for robot navigation. In IEEE International Conference on Robotics and Automation, pages 10608-10615. IEEE, 2023.
[18] Krishna Murthy Jatavallabhula, Ali Kuwajerwala, Qiao Gu, Mohd. Omama, Tao Chen, Shuang Li, Ganesh lyer, Soroush Saryazdi, Nikhil Varma Keetha, A. Tewari, J. Tenenhaum, Celso M. de Melo, M. Krishna, L. Paull, F. Shkurti, and A. Torralba. Conceptfusion: Open-set multimodal 3d mapping. arXiv.org, 2023.
[19] Baoxiong Jia, Yixin Chen, Huangyue Yu, Yan Wang. Xuesong Niu, Tengyu Liu, Qing Li, and Siyuan Huang. Sceneverse: Scaling 3d vision-language learning for grounded scene understanding. arXiv preprint vrXiv:2201.09246 2024.
[20] Kacper Kania, Kwang Moo Yi, Marek Kowalski, Tomasz Trzcińskii and Andrea Tagliasacchi. Conerf: Controllable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 06623-18622, 2022.
[21] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4). July 2023.
[22] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4):139-1 2023.
[23] Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf' Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19729-19739, 2023.
[24] Chung Min Kim, Mingxuan Wu, Justin Kerr, Ken Goldberg, Matthew Tancik, and Angjoo Kanazawa. Garfield: Group anything with radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 11530-215333 2024.
[25] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitchead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[26] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf editing via feature field distillation. Advances in Neural Information Processing Systems, 25:23311-23332 2022.
[27] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiV, 2023.
[28] Verica Lazova, Vladimir Guzov, Kyle Olszewski, Sergey Tulyakov, and Gerard Pons-Moll. Control-nerf: Editahle feature volumes for scene rendering and manipulation. In Proceedings of the IEEF/CVF Winter Conference on Applications of Computer Vision, pages 4340-4350, 2023.
[29] Chengshu Li, Ruohan Zhang, Josiah Wong, Cem Gokmen, Sanjana Srivastava, Roberto Chen Wang. Gabrael Levine, Wensi Ai, Benjamin Martinez, al. Behavior- lk: A human-centered, embod- ied ai benchmark with 1,000 everyday activities and realistic simulation. arXiv preprint 2rX 2024.
[30] Ruilong Li. Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner. Tava: Template-free animatable volumetric actors. In European Conference on Computer Vision, pages 419 9934996..66 Springer, 2022.
[31] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove. Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1521-5531, 2022.
[32] Bangyan Liao, Delin Qu, Yifei Xue, Huiqing Zhang, and Yizhen Lao. Revisiting rolling shutter bundle adjustment: Toward accurate and fast solution. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4863 4871, June 2023.
[33] Jingbo Zhang3 Zhihao Liang4 Jing Liao, Yan-Pei Cao, and Ying Shan. Advances in 3d generation: survey. arXiv preprint arXiv:2401. 17807, 2024.
[34] Kunhao Liu, Fangneng Zhan, Jiahui Zhang, Muyu Xu, Yingchen Yu, Abdulmotaleb El Saddik, Christian Theobalt, Eric Xing, and Shijian Lu. 3d open- vocabblayy segmentation with foundation models. arXiv preprint arXi::2005.14093 2(3)66, 2023.
12

---

[35] Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero- 1-to-3: Zero-shot one image to 3d object. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9298-9309, 2023.
[36] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.097 09713, 2023.
[37] Haimin Luo, Min Ouyang, Zijun Zhao, Suyi Jiang, Longwen Zhang. Qixuan Zhang, Wei Yang, Lan Xu, and Jingyi Yu. Gaussianhair: Hair modeling and rendering with light-aware gaussians. arXiv preprint arXiv:2402. 10483, 2024.
[38] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- 106, 2021.
[39] T. Müller, Alex Evans, Christoph Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM Transactions on Graphics. 2022.
[40] M. Oquab, Timoth' ee Darcet. T. Moutakanni, Huy Q Vo. Marc Szafraniec. Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Mahmoud Assran, Nicolas Ballas, Wajeiech Galuba, Russ Howes, Po- Yao (Bernie) Huang, Shang- Wen Li, Ishan Misra, Michael G. Rabhat, Vasu Sharma, Gabriel Synnaeve, Huijiao Xu, H. Jégou, J. Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision. arXiv.org, 2023.
[41] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Bruuala Nerties: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[42] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Bruulla and Steven M. Seitz. Hypemerf: A nigher-dimensione representation for topologically varying neural radiance fields. ACM Trans. Graph., 40(6), dee 2021.
[43] Minghan Qin, Wanhua Li, Jiawei Zhou, Haoqian Wang, and Hanspeter Pfister. Langsplat: 3d language gaus- sian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition. pages 20051-20066 2024.
[44] Delin Qu, Chi Yan, Dong Wang, Jie Yin, Qizhi Chen, Dan Xu, Yiting Zhang, Bin Zhao, and Xuclong Li. Implicit event-rgbd neural slam. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 19584-19594, June 2024.
[45] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, 8763. PMLR, 2021.
[46] Nur Muhammad (Mahi) Shafiullah, Chris Paxton, Lerrel Pinto, Soumith Chintala, and Arthur D. Szlam. Clip-fields: Weakly supervised semantic fields for robotic memory. ArXiv, 2022.
[47] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang. and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, 2023.
[48] tin-Chuan Shi, Miao Wang, Hao-Bin Duan, and Shao-Hua Guan. I Language emhedded 3d gaussians for open- neen-vocabulary scene understanding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 5333-5343, 2024.
[49] Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5).2732 27222 2023.
[50] Jonathan Steuer. Defining virtual reality: dimensions determining telepresence. 1992.
[51] Matthew Tancik, Fthan Weber, Evonne Ng, Ruilong Li, Brent Yi, Kerr, Terrance Wang, Alexander Kristoffersen, Jake Austin, Kamyar Salahi, Abhik Ahuja, David McAllister, and Angjoo Kanazawa. Nerfstudio: A modular framework for neural radiance field development. ArXiv, 2023.
[52] Jiaxiang Tang, Jiawei Ren, Hang Zhou, Ziwei Liu, and Gang Zeng. Dreamgaussian: Generative gaus- sian splatting for efficient 3d content creation. In The Twelfth International Conference on Learning Representations, 2024.
13

---

[53] Yiwen Tang, Ray Zhang. Jiaming Liu, Zoey Guo. Bin Zhao, Zhigang Wang. Peng Gao, Hongsheng Li, Dong Wang. and Xuelong Li. Any2point: Empowering any-modality large models for efficient 3d understanding. In European Conference on Computer Vision, pages 456 Springer. 2025.
[54] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[55] Vadim Tschernezki, Tro Laina, Diane Larlus, and Andrea Vedaldi. Neural feature fusion fields: 3d distillation of self-supe 2d image representations. In International Conference on 3D Vision, pages 443-453. IEEE, 2022.
[56] Feng Wang, Zilong Chen, Guokang Wang. Yafci Song, and Huaping Liu. Masked space- time hash encoding for efficient dynamic scene reconstruction. Advances Neural Information Processing Systems, 36, 2024,
[57] Feng Wang. Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 19706-19777 2023.
[58] Zian Wang, Tianchang Shen, Merlin Nimier-David, Nicholas Sharp, Jun Gao, Alexander Keller, Sanja Fidler, Thomas Muller, and Zan Gojeic. Adaptive shells for efficient neural radiance field rendering. ACM Transactions on Graphics, 42:1-15, 2023.
[59] Guanjun Wu. Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-tin time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 20310-20000 20320, 2024.
[60] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[61] Chi Yan, Delin Qu, Dan Xu, Bin Zhao, Zhigang Wang, Dong Wang, and Xuelong Li. Gs-slam: Dense visual slam with 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19595-19660 June 2024.
[62] Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for lighltidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20331-20341, 2024.
[63] Heng Yu. Joel Julin, Zoltán Á Milacski. Koichiro Ninumaa, and László A Jeni. Cogs: Controllable gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2024.
[64] Heng Yu. Koichiro Nüiuuma, and László A Jeni. Confies: Controllable neural face avatars. In 2023 IEEE 17th International Conference on Automatic Face and Gesture Recognition (FG), pages 1-8. IEFF, 2023.
[65] Wentao Yuan, Zhaoyang Lx, Tanner Schmidt, and Steven Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedingy of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13144-13152. 2021.
[66] Raza Yunus, Jan Eric Lenssen, Michael Niemeyer, Yiyi Liao, Christian Rupprecht, Christian Theobalt, Gerard Pons-Mote Moll, Jia-Bin Bin Huang, Vladislav Golyanik, and Eddy llg. Recent trends in 3d reconstruction of general non- rrgid scenes. In Computer Graphics Forum, page el e15062. Wiley Online Library, 2024.
[67] Hao Zhang, Fang Li, and Narendra Ahuja. Open-nerf: Towards open vocahulary nerf decomposition. In Proceedings of the IEFF/CVF Winter Conference on Applications of Computer Vision, pages 3456-3465, 2024.
[68] Chengwei Zheng, Wenbin Lin, and Feng Xu. Editablenerf: Editing topologically varying neural radiance fields by key points. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 8317-8327, 2023.
[69] Shuaifeng Zhi, Tristan Laidlow, Stefan Leutenegger, and Andrew Davison. In-place scene labelling and understanding with implicit scene representation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15838 15838-15847, 2021.
14

---

[70] Ce Zhou, Qian Li. Chen Li, Jun Yu. Yixin Liu, Guan Wang, Kaichao Zhang, Cheng Ji, Qi Yan. Lifang He, Hao Peng, Jianxin Li. Jia Wu, Ziwei Liu, P. Xie. Caiming Xiong, Jian Pei. Philip S. Yu, Lichao Sun Michigan State University, University, Lehigh University, M. University, Nanyang Technological University, University of California at San Diego, D. University, U. Chicago, and Salesforce Al Research. A comprehensive survey pretrained foundation models: A history from bert to chatgpt. ArXiv, 2023,
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Both the abstract and introduction accurately reflect the contributions and scope LiveScene.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of work in the conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
16

---

Justification:
The paper includes theoretical results, assumptions, and proofs, additional details can be found in the supplemental material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide detailed information on the experimental setup and results the main paper and supplemental material, and provide link to our project.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
17

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We will provide open access to the data and code, and detailed instructions to reproduce the main experimental results. But in the review process, we only provide anonymous link to our project.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
public/ CodeeSuuu ss ssionPolicyy for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips cc/ httpss/ccccbbcccccccccccccccc wwabbbic/uuiiii ides/ CoodeS ubmi ss ssssssinn ieriionollii connolccy icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permirted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide detailed information on the experimental setup and results in the main paper and supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not report error bars or statistical significance in the paper, but the results are reproducible and the code will be released. Besides, we provide average results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed information on the experimental setup and results in the main paper and supplemental material.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http://pps////e.....././///aaaaa
Answer: [Yes]
Justification: we have reviewed the NeurIPS Code of Ethics and believe that research conforms to
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: We focus on the interactive scene reconstruction and manipulation task, and societal impacts can be ignored.
Guidelines:
19

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: We focus on the interactive scene reconstruction and manipulation task, is safe for release.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We use existing datasets and models, and properly credit the original owners and respect the licenses. Besides, we reproduce the results with the released code, data and paper.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
20

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, (papersw saperswitheock com/ datase has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We contribute new dataset and code, and provide detailed documentation in the supplemental material. In the future, we will release the dataset and code with clear documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
21

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "Yh8QfqftBt", "reviewer": "Reviewer_PajG", "summary": "This paper proposed LiveScene, a NeRF-based approach to enable indoor-scale controllable scene reconstruction and novel view synthethis. By extending K-Panes with a object-aware multi-scale space factorization, scene-level 3D space with articulated objects could be modeled with motion patterns via densely collected monocular video with camera poses. On exsiting benchmark datasets and newly proposed datasets OmniSim/InterReal, the porposed method LiveScene achieves the best overall performance.", "questions": "I think overall this paper is quite interesting and effective. I would  like to see a more diverse demos to highlight its streagths and more clarifications on the detail of this paper. In initially the paper  is a bit hard for me to clearly follow. Therefore, in additio to the concerns raised in the weaknesses section, I would encourage the authors to improve the organization by clearly describing the pipeline details and improve the correspondence between text and figures.", "limitations": "Limitations have been partly addressed in the main paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The overall method is well motivated to address the more challenging indoor-scene level controllable NeRF. The introduction of control variables and their space spaces make the overall training feasible.\n\n- The extensive experiments as well as demos proves the effectiveness of LiveScene, both quanlitatively and quantitaitively.", "weaknesses": "Though the overall resutls seem promising, I still have several concerns regarding the formulation and lack of clarifications in certain key aspects.\n\n- The only additional attribute of overll space is modelled as a 3+alpha dimension space, how to cope with time variations? Are the time dimension implicitly encoded within the contro variables to cope with motions (such as open/close the door)? Or is there any explicit formulation of time dimension?\n\n- What is the potential maximum number of objects within the scene? And what is the potential limitations when scaling up to more objects? In Tab.6 of supplement, 6 objects at most is validated. How about more diverse objects?\n\n- Is it possible to encode more complex or fine-grained object control (e.g.,open left side door of a double-open fridge), especially when the training data mainly contains the fully open and fully-close state. Specifically, I am wondering what is the interpolation capability of the propsoed interaction-aware feature space and its generalization capability to unseen but correlated states.\n\n- As mentioned in Appdix D, ‘Interaction Variable MSE’ mentions that the interaction values are fully supervised and GT labels are also used during inferenced to enble control. It would be good to see, in practical cases, without GT interaction values, what is the performance degradations, which could further strengthen the potential applciations and reveal potential limitaions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**#Q1. How to cope with time variations? Are the time dimension encoded within control variables?**\n\nYes. The timestep variables, 3D features, and interaction variables are hybridized and fed into the interaction probability decoder to yield the probability distribution in implementation, as shown in **Fig.2 of the manuscript**. It’s critical to distinguish between 4D reconstruction and our interactive scene modeling. Unlike 4D reconstruction, where all properties synchronously vary over time, interactive scene modeling involves independent changes to individual objects, which is precisely why we require additional $\\alpha$ dimensions to model the scene. Hence, each object forms its own local 4D scene, where interactions within a local scene are manipulated through interaction variables $\\boldsymbol{\\kappa}$. \n\n**#Q2. Potential maximum number of objects and limitations** **within the scene.**\n\nIn the following table and **Fig.1(b) of the Attached PDF,** we validate the performance of LiveScene in scenarios with up to 10 complex interactive objects. Notably, our method demonstrates robustness in rendering quality, which does not degrade significantly as the object number increases. The number of objects is not a major limiting and our method is still feasible as long as the dataset provides mask and control variable labels. In contrast, the occlusion and topological complexity between objects do affect the reconstruction results, which will be discussed in the limitations section.\n\n|  object number scaling    | #2 | #4 | #6 | #8 | #10 |\n| --- | --- | --- | --- | --- | --- |\n| PSNR | 34.19 | 33.39 | 32.64 | 32.63 | 32.09 |\n| SSIM | 0.958 | 0.950 | 0.948 | 0.948 | 0.936 |\n| LPIPS | 0.081 | 0.077 | 0.093 | 0.104 | 0.110 |\n\n**#Q3. Is it possible to control more complex or fine-grained objects? The interpolation and generalization capability of the interaction-aware feature.**\n\nYes. In **Fig.1(c) of the Attached PDF**, we demonstrate the fine-grained control capability of LiveScene on a refrigerator and cabinet dataset without part-based labels. Our method can control a part of the object even though there are no individual part-based interaction variable labels. However, the effect is not entirely satisfactory, due to the lack of labels and CLIP's limited understanding of spatial relationships.\n\nAdditionally, we conducted an experiment to examine the method's interpolation capability for unseen yet correlated states. As shown in **Fig.2 of the Attached PDF,** we mask the camera view and control variable labels from 100% to 30% to increase the unseen yet correlated states. The results show that our method achieves good interpolation and generalization performance, as the image rendering quality remains stable between 100% and 40%. However, the algorithm deteriorates and causes artifacts, as the perspective and labels are increasingly missing in the last column of the table and picture in **Fig.2 of the Attached PDF**. Hence, the proposed feature interpolation can only maintain interaction and view consistency but does not completely address the extreme view missing issue. We’ll clarify this in the limitation section of the revised version.\n\n**#Q4. GT labels during inference control and performance degradations without GT interaction values.**\n\nThe interactive variable labels are only required during training, not during inference control. As shown in **Fig.2 of the Attached PDF**, the decrease in interactive variable density only starts to have an adverse impact once it reaches a certain threshold (40%~30%), which has been clarified in the **limitation Sec.6 of the manuscript**. Additionally, we provide an experiment in **Fig.3 of the Attached PDF** which compares the rendering results with and without GT interaction variables. Note that we only mask the GT interaction variables but provide the supervision of RGB in both setting. According to the results, even without the interaction variable supervision, the proposed method can still achieve satisfactory rendering quality (31.45 vs 31.58 in PSNR) but loses the ability to control. Specifically, our method is unable to open the dishwasher and degenerates into 4D reconstruction without any interaction variable supervision. We have carefully revised our manuscript and will provide more detailed experiments to illustrate.\n\n**#Q5. More diverse demos to highlight and more clarifications on details of the paper.**\n\nThanks for your kind comment. We have provided a project page in **supplemental Sec.C** with an anonymous link to demonstrate the interactive scene reconstruction and multimodal control capabilities. Please refer to the anonymous link for more information. Besides, we have carefully revised our manuscript to eliminate all typos and presentation issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LA0d49r1sn", "reviewer": "Reviewer_aZ7V", "summary": "This paper tackles an important problem in reconstructing interactive 3D scenes with language grounding. The authors proposed to use object-based modeling of different deformation fields over the dynamic NeRF pipeline and equip it with language embeddings for grounding interactions. The authors constructed two synthetic datasets OmniSim and InterReal for data collection and evaluations. Experimental results show that their methods significantly outperform prior methods on reconstruction and language grounding.", "questions": "See the weakness section.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The problem of reconstructing interactive scenes with language grounding is an important topic to embodied AI, we have seen many related works that boost the development of robot perception.\n- The construction of OmniSim and InterReal datasets could be beneficial for research in dynamic reconstruction and robotics.\n- The authors showed significant performance improvement on their constructed dataset, outperforming existing articulated object reconstruction models by a large margin.", "weaknesses": "- Despite the good motivation, one major concern about its paper is its poor presentation in terms of notations and details. Several key detail designs are omitted in both Sec.3/4 and supplementary.\n  - How are disjoint regions $\\mathcal{R}$ defined? it seems from the start of the description that this knowledge is already given. How do we determine the number of subregions? Any prior used for learning the deformation in each subregion? Since jointly optimizing the belonging relationship of each point to each region and the deformation for each field is pretty difficult as far as I know. \n  - The notations used are confusing, especially in Sec.3.2 which is an important portion of text to help understand the methodology. In Eq.3, what does $\\Theta$ mean? It is different from the $\\Theta$ in Eq.2 but is it just another MLP prediction? how to determine them for each region?\n  - As for the dataset curation, in each data sample will there be multiple objects being interacted (since you modeled many sub-deformation fields)?\n  - The authors should mention if any priors were added to the implemented baselines as well because methods like CoGS were not originally designed to handle multiple objects if I'm understanding it correctly.\n- The current dynamic reconstruction model still stays at the rendering level, some explorations on extending it to 3D meshes or simulated environments might provide more insights on using this model for future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive and voluntary review suggestions in both methodology and writing. We have carefully reviewed and corrected the entire manuscript, striving to eliminate any organizational and typos issues. Sincerely hope the proposed method and dataset in this paper will contribute to the field of interactive scene reconstruction.\n\n**#Q1. The definition and determinant of disjoint regions 𝑅. How to determine the number of subregions?**\n\nAs shown in **Fig.3(b) of the manuscript**, the basic idea of disjoint regions is that we divide a complex interactive space into regions, namely disjoint regions 𝑅, each with an independent local deformable field, where interactions within the local field are manipulated through interaction variables. As the reviewer pointed out, jointly optimizing to form regions is challenging; hence, we utilize mask supervision and focal loss $\\mathcal{L}_\\text{focus}$ in **Eq.7** to segment regions during training. \n\n$\\mathcal{L} _ \\text{focus} = \\beta \\cdot \\left(1 - e^{\\sum _ {i=1}^{\\alpha} \\tilde{\\mathbf{y}} _ {m}^i \\log(\\mathbf{q} _ i)}\\right)^{\\gamma} \\cdot \\left(- \\sum _ {i=1}^{\\alpha} \\tilde{\\mathbf{y}} _ {m}^i \\log(\\mathbf{q} _ i)\\right)$, \n\nwhere $\\tilde{\\mathbf{y}}_{m}^i$ is the ground truth label, $\\mathbf{q}_i$ is the predicted probability $\\Theta(\\boldsymbol{\\kappa}, \\boldsymbol{\\theta}_s(\\mathbf{x}))$ rendering from the interactive probability field, $\\beta$ is the balancing factor, and $\\gamma$ is the focusing parameter. \n\nThe **number, shape, and density** of regions are gradually established in training by maximizing the probability distribution outputs of the interaction probability decoder shown in **manuscript Fig.2**. We provide additional experiments in **Fig.4 of the Attached PDF** to illustrate the learning process of disjoint regions from 0 to 1000 training steps. The results demonstrate a clear trend that, as training advances, the proposed method is able to progressively converge to the vicinity of the interactive objects, thereby establishing interactive regions. Furthermore, we provide interaction feature visualization of  x-$\\boldsymbol{\\kappa}$, y-$\\boldsymbol{\\kappa}$, and z-$\\boldsymbol{\\kappa}$ in **Fig.1(a) of Attached PDF** to illustrate latent feature distribution. It can be seen that the features are clustered around the spatial coordinates of interactive objects, corresponding to the disjoint regions 𝑅.\n\n**#Q2. What does Θ in Eq.3 mean? Presentation of notations and paper organization.**\n\n $\\Theta$ in Eq.3 represents the projection operation in Eq.2 to map the interaction variables $\\boldsymbol{\\kappa}$ to the most probable cluster region. It is implemented with the interaction probability decoder (MLP with the plane feature) in **Fig.2 of the manuscript**. In addition, all the region is split by the probability distribution yielded with the only one interaction probability decoder according to Eq.3. The interaction probability decoder is primarily trained with the constraints of the mask (focal loss $\\mathcal{L} _ \\text{focus}$ in **Eq.7**) and RGB (rendering loss $\\mathcal{L} _ \\text{MSE}$ in **Eq.6**). We have carefully revised our manuscript to clarify the notations.\n\n**#Q3. Will there be multiple objects in each data sample?**\n\nYes. Our method and dataset are designed for complex scenes containing multiple objects. Please refer to the anonymous link in **supplemental Sec.C and Tab.5** for details.\n\n**#Q4. Any priors were added to the implemented baselines?**\n\nYes, we introduce extensions of K-Planes[1], namely MK-Planes and MK-Planes*, which enable their control capabilities by generalizing them from $C^2_4$ planes to $C^2_{3+α}$ and $3+3α$ planes, as elaborated in **Sec.5 of the manuscript**. In addition, CoGS[2] is able to control multiple object elements according to the original paper. We emailed authors and extended the CoGS based on Deformable Gaussian[3] to tackle multiple interactive objects. We’ll clarify this in the revised version.\n\n**#Q5. Future explorations on extending it to 3D meshes or simulated environments.**\n\nThanks. Building interactive simulations from real-world scenarios holds great promise, particularly in Embodied AI. In the future, we will further explore this area, including explicit scene representation such as 3DGS and Mesh, as well as interactive generation, .etc.\n\n**Reference**\n\n[1] K-Planes: Explicit Radiance Fields in Space, Time, and Appearance, CVPR2023\n\n[2] CoGS: Controllable Gaussian Splatting, CVPR2024\n\n[3] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction, CVPR2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RaivN9iY70", "reviewer": "Reviewer_a4DY", "summary": "The paper addresses the complex challenge of reconstructing and controlling multiple interactive objects in complex scenes from monocular videos without prior modeling of geometry and kinematics. This task is critical for advancing fields like virtual reality, animation, and robotics, where understanding and interacting with 3D environments are essential. The proposed framework decomposes interactive scenes into local deformable fields. This factorization allows for precise modeling of individual objects’ interactions. Additionally, a multi-scale interaction probability sampling strategy is introduced to accurately sample interaction-relevant 3D points within these fields, enabling effective control over object dynamics in complex environments. The interaction-aware language embedding method generates dynamic language embeddings that adapt to varying interaction states. This allows users to control interactive objects through natural language commands, enhancing the interface's intuitiveness and accessibility. Authors also contributed OmniSim and InterReal datasets. These datasets are the first to offer scene-level physical interaction data, comprising 28 scenes with a total of 70 interactive objects. They provide a valuable resource for evaluating the performance of interactive scene modeling methods.", "questions": "What's the memory cost of the proposed method? What's the finest granularity of interaction can it handle (i.e. operate on a very thin/small object)? How does the model handle repetitive objects in the scene? In Supp. Fig16, the language query is \"top cabinet\", but it seems the model finds the microwave?", "limitations": "Authors discussed limitations in terms of closed vocabulary, caused by OpenCLIP. Given that authors did not show much real-world scene manipulation results, potential limitations in real-world scenes should also be discussed as such scenes are in general more complicated. No societal impact was discussed.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The paper is well motivated and sets a clear difference from previous works. Results look good and promising. Figures and illustrations are helpful and informative.\n+ The factorization technique that decomposes complex scenes into local deformable fields allows for more granular control and precise modeling of individual object interactions within a complex scene, addressing the high-dimensional challenge that previous methods struggled with.\n+ By embedding language control within the interactive radiance fields, the framework allows users to manipulate and interact with 3D environments using simple language commands, greatly enhancing user accessibility and interaction fidelity.\n+ Authors provided abundant demos on their project page, which is helpful", "weaknesses": "- Might need to slightly enlarge texts in figures.\n- Not necessarily a weakness but authors can consider visualize some latent features (instead of illustrations like in fig2-4) to better show the decomposition of the high-dimensional feature space.\n- Lack some qualitative results on real-world dataset and existing public dataset. Also, in the only InterReal qualitative results (fig.11), k-planes results were missed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for reviewing our paper and for the valuable feedback! \n\n**#Q1. Enlarge texts in figures.**\n\nThanks. We have carefully revised the manuscript according to your comments.\n\n**#Q2. Visualization of some latent features.**\n\nWe provide additional interaction feature visualization of  x-$\\boldsymbol{\\kappa}$, y-$\\boldsymbol{\\kappa}$, and z-$\\boldsymbol{\\kappa}$ in **Fig.1(a) of Attached PDF** to illustrate latent feature distribution. It can be seen that the features are clustered around the spatial coordinates of interactive objects, corresponding to the local deformable fields in **Sec 3.2 of the manuscript.**.\n\n**#Q3. More qualitative results on real-world and public datasets. K-Planes results were missed.**\n\nTo our knowledge, existing view synthetic datasets for interactive scene rendering are primarily limited to a few interactive objects falling short of scene-level interactive reconstruction. **Fig.5 of the Attached PDF** provides visualization comparisons with CoNeRF[1] and CoGS[3] in CoNeRF Controllable dataset. As the results show, LiveScene outperforms existing SOTA methods for interactive rendering, achieving higher-quality rendering. More detailed experiments will be provided in the revised version. We must emphasize that K-Planes[2] was originally designed for 4D reconstruction and thus it is not suitable and unfair for controllable rendering. Meanwhile, our enhanced baselines, MK-Planes and MK-Planes* in **Line 218 of Sec.5 of the manuscript.**, require dense interactive variable inputs, which are unavailable in the InterReal and CoNeRF Controllable datasets, making the comparison with K-Planes infeasible.\n\n**#Q4. Memory cost of the proposed method**\n\nAs shown in Table 1, we report GPU memory usage in the **seq002_Rs_int** sequence of the OminiSim dataset with official parameter settings. Notably, LiveScene (w/o language grounding) requires approximately 8G GPU memory, which is lower than that of the other methods.\n\n| Method |  Batch Size | Ray Samples | Runtime Memory (A100) |\n| --- | :---: |:---:|:---:|\n| CoNeRF | 1024 | 256 | 71931 MiB |\n| MKPlane | 4096 | 48 | 12781 MiB |\n| MKPlane* | 4096 | 48 | 12185 MiB |\n| CoGS | 512 | —— | 25505 MiB |\n| LiveScene w/o lang | 4096 | 48 | 8441 MiB |\n\n**#Q5. The finest granularity of interaction can it handle.**\n\nThe minimum granularity of controllable objects is hard to measure, as it depends on scene complexity, object number, and camera view. Our method still achieves precise control in extreme cases, such as the chest in the **anonymous link video demos**. In this work, we primarily focus on joint objects in indoor scenes, and finer-grained control is still feasible as long as the dataset provides relevant mask and control variable labels*.\n\n**#Q6. Handle repetitive objects.**\n\nGood question. In fact, due to the limitations of CLIP's spatial relationships understanding ability, our method does not perform well in distinguishing between repeated objects. This limitation is a commonality among many 3D vision-language field methods, such as LERF[4] and OpenNeRF[5]. We’ll clarify this in the revised version.\n\n**#Q7. Annotation error in Supp. Fig16.**\n\nThanks. It’s a typo, and the correct language query is \"microwave\".\n\n**#Q8. More discussion of real-world potential limitations and societal impact**.\n\nBeyond the existing discussions in **Sec.6 of the manuscript**., a known limitation in real-world scenarios is that occlusion between objects can affect the interactive rendering effect. Besides, our method currently requires dense GT control variable annotation, which can be time-consuming and labor-intensive to obtain in real-world scenarios. We plan to explore sparse GT input methods (e.g., 3-frame annotation) to improve efficiency.\n\nRegarding social impact, our method is committed to building interactive simulators from real-world scenarios, providing real-to-sim interactive environments for Embodied AI, e.g. navigation, grounding, and action. However, as with all work that enables editable models, our method has the potential to be misused for malicious purposes such as deep fakes. We’ll clarify this in the revised version.\n\n**Reference**\n\n[1]  CoNeRF: Controllable Neural Radiance Fields, CVPR2022\n\n[2] K-Planes: Explicit Radiance Fields in Space, Time, and Appearance, CVPR2023\n\n[3] CoGS: Controllable Gaussian Splatting, CVPR2024\n\n[4] LERF: Language Embedded Radiance Fields, ICCV 2023\n\n[5] OpenNeRF: OpenSet 3D Neural Scene Segmentation with Pixel-Wise Features and Rendered Novel Views, ICLR2024", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lo664LbOrL", "reviewer": "Reviewer_94zG", "summary": "They introduce a language-embedded \"interactive neural radiance field\" that efficiently reconstructs and controls multiple objects within scenes. Factorization decomposes the scene into more local fields that can achieve local deformation.", "questions": "\"As illustrated in Fig 2, interactive objects exhibit mutual independence and interaction features F_alpha unevenly distribute in the 3+alpha-dimensional interacive space \\mathcal X and ggregate into cluster centers...\"\nFigure 2 shows this? This was hard to parse and to see from figure 2. I guess what you're trying to say is isntead of storing the motion state variables at every x, you'd much rather define cluster centers . However, I don't understand why this means you can project the point in 3+alpha-space dowen to a 4-dimensional space.", "limitations": "No issues here.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "My sense is that the technical novelty of this paper is high, though I'm not an expert in this domain. Additionally the evaluation seems thorough and the method seems well-considered.", "weaknesses": "At times, I feel the the language of the methods section can be made a little clearer. I had trouble following the motivation for a lot of the design decisions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Sincerely appreciate for your kind comments and voluntary review suggestions! We have carefully reviewed and corrected the entire manuscript to improve the paper's organization and presentation.\n\n**#Q1. Improve the presentation of the methods section**\n\nThanks! We have carefully revised the manuscript to eliminate presentation issues.\n\n**#Q2. Explain the Interaction Space Factorization**\n\nPlease refer to **Fig.2 and Sec.3.2 of the manuscripts**, where we decode and render the interactive space using spatial coordinates $(x,y,z)$ and interactive variables $(\\kappa_{1},\\kappa_{2},...,\\kappa_{\\alpha})$ as inputs. The basic idea is to divide the $3+\\alpha$ space into $\\alpha$ independent local deformable field, each of which interacted with the 4 coordinates $(x,y,z,\\kappa_{d})$. Specifically, we use an interaction probability decoder $\\boldsymbol{\\Theta}$ (MLP with the plane feature) In Fig.2 to predict the probability distribution of the ray samples, thereby determining the local deformable field region to which the sampling points belong. In the interactive ray sampling, each sample is associated with a distinct time coordinate and interaction variable. Therefore, through probability maximization, we can convert this $3+\\alpha$-dimensional sampling into a 3+1-dimensional sampling, wherein the index $d$ of interaction variables is derived by maximizing the probability distribution of decoder $\\Theta$. In this way, we map the interaction variables $\\boldsymbol{\\kappa}$ to the most probable cluster region $\\mathcal{R} _ {d}$ in the 4D space and accomplish the \"projection\":\n\n$d = \\text{argmax} _ {i} \\[ \\Theta(\\boldsymbol{\\kappa}, \\boldsymbol{\\theta} _ s(\\mathbf{x})) _ I \\]$,\n\nwhere $\\boldsymbol{\\theta} _ s(\\mathbf{x})$ is the probability features at position $\\mathbf{x}$ from 3d feature planes.\n\nAdditionally, we provide a more detailed figure to illustrate latent feature distribution in **Fig.1(a) of the Attached PDF** and the learning process of the local deformable field from 0 to 1000 training steps in **the Attached PDF Fig.4**. Please refer to the PDF for details.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank reviewer #**PajG**, #**aZ7V,** #**a4DY, and #94zG** for their thoughtful and constructive comments and suggestions. We have carefully revised our manuscript according to their comments. An **attached one-page PDF** is provided to show additional experiments and can be summarized as:

- Visualization of interaction feature planes x-$\boldsymbol{\kappa}$, y-$\boldsymbol{\kappa}$, and z-$\boldsymbol{\kappa}$ to illustrate the interaction feature distribution.
- A complex scenario illustration with up to 10 complex interactive objects.
- Part-level controlling experiment to show the fine-grained controlling ability.
- Rendering quality and visualization as the supervision of view and control variables decreases from 100-30% to show the effectiveness of feature interpolation.
- Rendering results with and without GT interaction variables.
- Visualization of disjoint regions (local deformable field) learning process from 0 to 1000 training steps.
- More View Synthesis Comparison results on the public CoNeRF-Controllable dataset.

In the original supplemental materials (**supplemental Sec.C**), We provide an **anonymous project page** to demonstrate the detailed datasets, interactive scene reconstruction, and multimodal control capabilities of LiveScene. Please refer to the anonymous link for more information.

Meta-review: The paper received very positive scores from the reviewers. They appreciated the problem, the proposed datasets, and performance improvements done in this work. The authors addressed most of the limitations during the discussion period. AC agrees and recommends acceptance. Congrats!
Decision: Accept (poster)