OmniRe: Omni Urban Scene Reconstruction
Authors: Ziyu Chen
Jiawei Yang
Jiahui Huang
Riccardo de Lutio
Janick Martinez Esturo
Boris Ivanovic
Or Litany
Zan Gojcic
Sanja Fidler
Marco Pavone
Li Song
Yue Wang
Venue: ICLR 2025 Spotlight (2025)
Keywords: Gaussians Splatting
Neural Rendering
Dynamic Scene Reconstruction
Autonomous Driving
Primary area: applications to robotics, autonomy, planning

Abstract: We introduce OmniRe, a comprehensive system for efficiently creating high-fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hindering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-length reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and constructs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. OmniRe allows holistically reconstructing any dynamic object in the scene, enabling advanced simulations (~60 Hz) that include human-participated scenarios, such as pedestrian behavior simulation and human-vehicle interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations on the Waymo dataset show that our approach outperforms prior state-of-the-art methods quantitatively and qualitatively by a large margin. We further extend our results to 5 additional popular driving datasets to demonstrate its generalizability on common urban scenes. Code and results are available at [omnire](https://ziyc.github.io/omnire/).

Published as a conference paper at ICLR 2025
OMNIRE: OMNI URBAN SCENE RECONSTRUCTION
Ziyu Chen Jiawei Yang Jiahui Huang
Riccardo de Lutio
Janick Martinez Esturo² Sanja Fidlers.s
Boris Ivanovic
Or Litany 2,5 Zan Gojeic
Marco Pavone
Li Song'
Yue Wang
1Shanghai Jiao Tong University Technion UUniversity of Toronto "NVIDIA Research University of Southern California
*Stanford University
ABSTRACT
We introduce OmniRe, comprehensive system for efficiently creating high- fidelity digital twins of dynamic real-world scenes from on-device logs. Recent methods using neural fields or Gaussian Splatting primarily focus on vehicles, hin- dering a holistic framework for all dynamic foregrounds demanded by downstream applications, e.g., the simulation of human behavior. OmniRe extends beyond vehicle modeling to enable accurate, full-Iength reconstruction of diverse dynamic objects in urban scenes. Our approach builds scene graphs on 3DGS and con- structs multiple Gaussian representations in canonical spaces that model various dynamic actors, including vehicles, pedestrians, cyclists, and others. Omn i Re allows holistically reconstructing any dynamic object in the scene, enabling ad- vanced simulations (-60 Hz) that include pedestrian behavior simulation and human-vehiele interaction. This comprehensive simulation capability is unmatched by existing methods. Extensive evaluations the Waymo dataset show that approach outperforms prior sfaeeoo-thhe-as art methods quantitatively and qualitatively by a large margin. We further extend our results additional popular driving datasets to demonstrate its generalizability on common urban scenes. Code and results are available at omnire.
scenarios, such as
INTRODUCTION
Creating photorealistic digital twins of 4D real-world is valuable for enabling high-fidelity simulation, robust algorithm training and evaluation. As autonomous driving algorithms increasingly adopt end-to-end models, the need for scalable and domain-gap- saaaiinn-a---reee simulation environments, where these systems can be evaluated in closed-loop, is becoming more evident. While traditional artist-generated assets are reaching their limits in scale, diversity, and realism, recent advances in data-driven methods offer strong alternative by creating realistic digital twins directly from real-world sensor data. Indeed, neural radiance fields (NeRFs) (Mildenhall et al., 2020; Barron et al., 2021; Yang et al., 2023b; Guo et al., 2023; Yang et al., 2023a; Wu et al., 2023b) and Gaussian Splatting (GS) (Kerbl et al., 2023; Yan et al., 2024) have emerged as powerful tools for reconstructing 3D scenes with high levels of visual and geometric fidelity. However, accurately and holistically reconstructing dynamic urban scenes remains significant challenge, especially due to the diverse dynamic actors and their complex rigid and non-rigid motions in real-world environments.
Several works have already tried to tackle this challenge. Early methods typically ignore dynamic actors and reconstruct only static parts of the scene (Tancik al., 2022: Rematas et al., 2022; Guo et al., 2023). Subsequent works aim to reconstruct the dynamic scenes by either (i) modeling the scenes as a combination of a static and time-dependent dynamic neural field, where the static-dynamic decomposition is an emergent property (Yang al., 2023a; Turki et al., 2023), or (ii) building a scene graph, in which dynamic actors and the static background represented nodes and reconstructed in their canonical frame. The nodes of the scene graph are connected with edges that encode relative transformation representing the motion of each actor through time (Ost et al., 2021; Kundu al., 2022; Yang al., 2023b; Wu et al., 2023b; Tonderski
Brualla al., 2021;
Work done during research internship at University of Southem California. Ziyu Chen <ziyu.sjuu Saayyyujuu@gmail..oooo. gmail.com>> Yue Wang <yue. @usc.cdu>>

---

Published as a conference paper at ICLR 2025
(a) Scene Decomposition
Corner Case Handling
2
(c) Scene Simulation
Figure I: (a) Decomposition of different parts of scene. (b) Out-of-distributin categories that are overlooked by previous methods can be accurately handled by OmniRe. (c) Oe enables diverse applications including vehicle editing (c1, c2), human-vehicle interaction (c3), human behavior simulation (c4), etc.
et al., 2024; Fischer al., 2024b). But both approaches fall short of meeting the requirements for comprehensive and interactive digital twins: while providing a more general formulation, methods of (i) lack editability and cannot be directly controlled with classical behavior models. Previous methods following (ii) still focus primarily on vehicles, which can be represented as rigid bodies, thereby largely neglecting other vulnerable road users (VRUs) such as pedestrians and cyclists that are fundamental and critical in urban scene simulation.
To fill this critical gap, our work aims to model all dynamic actors, including vehicles, pedestrians, and cyclists, and many others, in manner that allows for interactive simulation. This leads to two primary challenges: (i) developing unified approach for modeling diverse non-rigid dynamic actors, given the wide range of non-rigid categories in real-world scenes; (ii) giving specific focus on humans, as their behavior critical for decision makinng in scenarios like driving, where pedestrian actions directly impact safety. Thus, precise joint-level reconstruction (Lei et al., 2023; Jiang et al., 2022; Kocabas et al., 2024) is crucial for fine control of human behavior the simulator. To address the specific challenge of modeling human actors, must consider several additional complexities. First, in-the-wild scenarios present significant challenges for capturing human motion dynamics due to unfavorable sensor observations and cluttered environments with frequent occlusions (Wang et al., 2024; Yang et al., 2021; Wang et al., 2023). Furthermore, reconstructing high-fidelity human appearance from sparse sensor data beyond mere geometry dynamics adds additional complexity. Lastly, interactions with large equipment, such as wheelchairs or strollers, which cannot be represented by explicit templates (e.g., SMPL), further complicate both appearance and geometry modeling.
To address these challenges, we propose an "omni" system capable of handling diverse actors for urban digital twins. Our method OmniRe efficiently reconstructs high-fidelity urban scenes that include static backgrounds, driving vehicles, and non-rigidly moving dynamic actors (see Fig. 1). Specifically, we construct a dynamic neural scene graph (Ost et al., 2021) based on 3I) Gaussian Splatting (Kerbl et al., 2023), with dedicated Gaussian representations for different kinds of dynamic actors in their local canonical spaces. In our framework, backgrounds and vehicles are represented as static Gaussians, while vehicles undergo rigid body transformations to simulate their motion over time. For non-rigid actors, we incorporate the SMPL model to enable joint-level control for pedestrians using dynamic Gaussians, as SMPL provides prior template geometry for 3DGS initialization and explicit control for modeling desired human behaviors, which advantageous for downstream simulation applications. To extract SMPL parameters for human motion modeling,
2

---

Published as a conference paper at ICLR 2025
designed novel human body pose estimation pipeline dedicated to driving logs with multi- camera setups and severe in-the-wild occlusions. For other template-less dynamic actors, we propose a shared deformation field approach in self-supervised manner. This framework enables unified representation of all non-rigid categories and achieves specialized joint-level control for pedestrians. Thus, Omn Re allows for accurate representation and controllable reconstruction of most objects of interest in the scene. Notably, our representation is directly amenable to behavior and animation models that are commonly used in AV simulation (e.g., Fig. I-(c)).
To summarize, we make the following contributions:
We introduce OmniRe, holistic framework for dynamic urban scene reconstruction that embodies the "omni" principle of dynamic category coverage and representation flexibility. Omn. iRe lever- ages dynamic neural scene graphs based on Gaussian representations to unify the reconstruction of static backgrounds, driving vehicles, and non-rigidly moving dynamic actors ($ 4). It enables high-fidelity scene reconstruction and centerdd simulations, such pedestrian behavior and human-vehicle interaction- interactonn-ccpabllitt capabilities unmatched by existing methods. ($ 5).
We address the challenges of modeling humans and other dynamic actors from driving logs such as occlusion, cluttered environments, and the limitations of existing human pose prediction models ($ 4.2). We demonstrate our method on six popular driving datasets to show its generalizability in urban driving scenes (project page). While our findings are based on AV scenarios, they can generalize to other domains.
We perform extensive experiments and ablations to demonstrate the benefits of our holistic frame- work. Omn Re achieves state-of-the-art performance in scene reconstruction and novel view synthesis (NVS), significantly outperforming previous methods in terms of full image metrics (+1.88 PSNR for reconstruction and +2.38 PNSR for NVS). The differences are pronounced for dynamic actors, such vehicles (+1.18 PSNR), and humans (+4.09 PSNR for reconstruction and +3.06 PSNR for NVS) (Tab. 1).
RELATED WORK
Dynamic Scene Modeling. Neural representations are dominating novel view synthesis (Mildenhall et al., 2020; Barron et al., 2022; 2021; Müller et al., 2022; Fridovich- Keil et al., 2022; Kerbl et al., 2023). These have been extended in different ways enable dynamic scene reconstruction. Deformation-busel approaches (Pumarola et al., 2020; Park et al., 2021a; Tretschk et al., 2021; Park al., 2021b; b; Cai et al., 2022) and recently DeformableGS (Yang al., 2023c) and (Wu et al., 2023a) propose to model dynamic scenes using a 3D neural representation for the canonical space, coupled with deformation network mapping time-dependent observations to canonical deformations. These are generally limited to small scenes with limited movement, making them inadequate for challenging urban dynamic scenes. Modulation -base techniques operate by directly feeding the image time- stamps (or latent codes) as an additional input to neural representation (Xian et al., 2021; Li et al., 2021; 2022; Luiten et al., 2024). However, this generally results in an underconstrained formulation, therefore requiring additional supervision, such as depth (Li et al., 2021) and optical flow (Xian et al., 2021), or multi-view inputs captured from synchronized cameras (Li et al., 2022; Luiten et al., 2024). DeNeRF (Wu et al., 2022) proposed to expand on this formulation by partitioning the scene into static and dynamic fields. Following this, SUDS (Turki et al., 2023) and EmerNeRF (Yang et al., 2023a) have shown impressive reconstruction ability for dynamic autonomous driving scenes. However, they model all dynamic elements using a single dynamic field, rather than modeling each separately, thus they lack controllability. limiting their practicality sensor simulators. Explicit decomposition of the scene into separate agents enables controlling them individually. These agents can be represented as bounding boxes in a scene graph as in Neural Scene Graphs (NSG) (Ost et al., 2021) that is widely adopted in UniSim (Yang et al., 2023b). MARS (Wu et al., 2023b), NeuRAD (Tonderski et al., 2024), ML-NSG (Fischer et al., 2024b) and recent Gaussian-based works StreetGaussians (Yan et al., 2024), DrivingGaussians (Zhou et al., 2023), and HUGS (Zhou et al., 2024). However, these approaches handle only rigid objects due to limitations of time-independent representations (Ost et al., 2021; Wu et al., 2023b; Yang et al., 2023b; Zhou et al., 2023; 2024; Yan et al., 2024; Tonderski et al., 2024; Fischer et al., 2024b) or limitations of deformation-baut 2023). A recent concurrent work Fischer et al. (2024a) also considers non-rigid modeling using deformation field, addressing a subset of the challenges in modeling holistic dynamics, but does not address fine-grained human models that allow flexible control. To address them, Cmn i Re proposes
techniques (Yang et al., 2023c; Huang et al.,
3

---

Published as a conference paper at ICLR 2025
Gaussian scene graph that incorporates various Gaussian representations for both rigid and non-rigid objects, providing extra flexibility and controllability for diverse actors.
Human Modeling, Human bodies have variable appearance and complex motions. calling for dedicated modeling techniques. NeuMan (Jiang et al., 2022) proposes to employ the SMPL body model (Loper et al., 2015) warp ray points reconstruction non-rigid human bodies and warrants fine control. Similarly, recent works such as GART (Lei al., 2023), GauHuman (Hu & Liu, 2023) and HumanGaussians (Kocabas al., 2024) have combined the Gaussian representation and the SMPL model. However, these methods are not directly applicable in-the-wile As for recovering human dynamics in driving scenes, Yang et al. (2021) focuses on shape and pose reconstruction for LiDAR simulation, while Wang et al. (2023; 2024) aim recreate natural and accurate human motion from partial observations. However, these methods focus solely on shape and pose estimation and are limited in appearance modeling. In contrast, our method not only models human appearance but also integrates this modeling within holistic scene framework, to achieve comprehensive solution. Urban scenes typically involve numerous pedestrians, with sparse observation, often accompanied by severe occlusion. We analyze these challenges in detail and address them in $ 4.2.
canonical space. This approach enables the
3 PRELIMINARIES
3D Gaussian Splatting. First introduced in Kerbl et al. (2023), 3D Gaussian Splatting (3DGS) represents scenes via set of colored blobs G = {g) whose intensity distribution Gaussian. Each Gaussian (blob) g = (o. μ.. q. s, c) contains the following attributes: opacity o E (0, 1), mean position μ E R rotation q IR represented as a quaternion, anisotropic scaling factors s RR.. and view-dependent colors c E R represented as spherical harmonics (SH) coefficients. To compute the color C' of pixel, Gaussians overlapping with this pixel are sorted by their distance to the camera center (sorted by i E NN) and a- a-blended: C' EieN C; [[;= (1-aj), az), where a; is computed as ag 02 exp(-5) - (p- 14:)² ",)' 'D - 14))). E; is the 2D projection covariance. We further define the application of rigid (affine) transformation T = (R, SE((3 all Gaussians in the set as: T & G = (o, R2 + t, Rot(R, q), 8, c), where Rot(.) denotes rotating the quaternion by the rotation matrix.
Skinned Multi-Person Linear (SMPL) Model. SMPL (Loper et al., 2015) is a parametric human body model that combines the advantages of a parametric mesh with linear blending skinning (LBS) to manipulate body shape and pose. At its core, SMPL uses a template mesh M₁ (V.F) defined in canonical rest pose, parameterized by n, vertices V E x3 The template mesh can be shaped
and transformed using shape parameters , and pose parameters O: Vs- = V + (B( + Br where (3) E R. x3 and (f) E TR th x3 are the zyz offsets individual vertices (Kocabas et al., 2024) and Vs are the vertex locations in the shaped space.
To further deform the vertices Vss to achieve the desired pose 0', SMPL utilizes pre-defined LBS weights W E x*. and the joint transformations G to define the deformation of each vertex 0; D2' = <Ex Gk) U. where ng is the number of joints, and the joint transformations G are derived from the source pose O, the target pose 0' and shape B. The pose parameters include the body pose component O, E 23333x3 and the global orientation component 0g E RR333 For more details of SMPL., we refer readers to Loper al. (2015). Our method obtains pose parameters 0 for each pedestrian across all frames, as well as their individual shape parameters /3 e _ these pose sequences initialize the non-rigid dynamics of pedestrians. The detailed process described in 4.2.
4 METHOD
As overviewed in Fig. 2, we build a comprehensive 3DGS framework that holistically reconstructs both the static background and diverse movable entities. We discuss our systematic approach that represents different semantic classes with diverse Gaussian representations in $ 4.1, highlighting that this complex yet efficient system-level framework is one of our primary contributions. Modeling humans in unconstrained environments is particularly challenging due to the complexity of human motions and the difficulty of accurately modeling geometry and appearance due to severe occlusions in the wild. We present our approach to this problem in $ 4.2, which significantly expands our
4

---

Published as a conference paper at ICLR 2025
Movable Entilies
(R. pol
time
SMPL Nodes
Detoraaton
Crisal
BBA
Rigid Nodes
Qquara
M/
Gaussian Socne Graph
SMPL
Nodes
Detommble Nodes
Oydiriodd
Aedestriias
redutrrian, Gyesers
Sharal
60
Detemutice
Static Background
Non-Rigid Modeling
Barkgroond Node
Sky Node
Scone Assets
Rendering
Supervision
Figure 2: Method Overview. Gaussians of all foreground models are defined in their local or canonical spaces. At given time the Gaussians are deformed and transformed into the world space, forming a Gaussian scene graph together with background Gaussians to model the entire scene. The Gaussians in the scene graph are rasterized to render images and depth, and are jointly optimized using reconstruction losses. We utilize SMPL Gaussians to model non-rigid human bodies and deformable Gaussians handle out-of-distributton non-rigid categories.
effectiveness in common driving scenes. Lastly, we show how the scene representation is end-to-end optimized to obtain faithful and controllable reconstructions in 4.3.
4.1 DYNAMIC GAUSSIAN SCENE GRAPH MODELING
Gaussian Scene Graph. To allow for flexible control of diverse movable objects in the scene without sacrificing reconstruction quality, opt for a Gaussian Scene Graph representation. Our scene graph is composed of the following nodes: (1) Sky Node representing the sky that is far away from the ego-car, (2) Background Node representing the static scene background such as buildings, roads, and vegetation, (3) set of Rigid Nodes, each representing rigidly movable object such as vehicle, (4) set of Non-rigid Nodes that model non-rigid individuals, e.g. pedestrians and cyclists. Nodes of type (2,3,4) can be converted directly into world-space Gaussians which we will introduce next. These Gaussians are concatenated and rendered using the rasterizer proposed in Kerbl al. (2023). The Sky Node is represented by an optimizable environment texture map, similar to Chen et al. (2023), rendered separately, and composited with the rasterized Gaussian image with simple alpha blending.
Background Node. The background node is represented by a set static Gaussians gov These Gaussians are initialized by accumulating the LiDAR points and additional points generated randomly in accordance with the strategy described in Chen et al. (2023).
Rigid Nodes. Gaussians representing the vehicles (e.g. cars or trucks) are defined as Grigid in the object's local space (denoted by the upper bar), where the index of the vehicle/node. While the Gaussians within a vehicle will not change over time in the local space, the positions of Gaussians in world space will change according to the vehicle's pose T, E (3). At given time t R, the Gaussians are transformed into world space by simply applying the pose transformation:
griedd (f) = T. (t) @ Greid
(I)
Non-Rigid Nodes. Non-rigid individuals are often overlooked by previous methods (Zhou et al., 2024; Yan et al., 2024; Zhou et al., 2023; Fischer al., 2024b) due to the modeling complexity, despite their importance for human-centered simulation. Unlike rigid vehicles, non-rigid dynamic classes such as pedestrians and cyclists, require extra consideration of both their global movements in world space and their continuous deformations in local space to accurately reconstruct their dynamics. To enable a reconstruction that fully explains the underlying geometry, we further subdivide the non-rigid nodes into categories: SMPL Nodes for walking running pedestrians with SMPL templates that enable joint-level control and Deformable Nodes for instances (such as cyclists and other template-less dynamic entities).
non-rigid
Non-Rigid SMPL Nodes. As introduced in $ 3, SMPL provides a parametric way of representing human poses and deformations, and hence use the model parameters (0(1), B) to drive the 3D)
5

---

Published as a conference paper at ICLR 2025
Gaussians within the nodes. Here (t) 24x3x3 represents the human posture that changes over time t. For each node indexed by h, We tessellate the SMPL template mesh M, instantiated from the resting pose (the 'Da' pose) with 3D Gaussians GMM.L using a strategy similar to Lei et al. (2023), where each Gaussian is binded to its corresponding vertex of Mr. The world-space Gaussians for each node can be then computed as:
(t) = (t) @ LBS (MM) M.M.).
Here (1) SI((3) is the global pose of the node time I, and LBS(-) is the linear blend skinning operation that deforms the Gaussians according to the SMPL pose parameters. In order to compute the LBS operator, one first precomputes the skinning weights of each Gaussian in GAMPL w.r.t. the SMPL key joints. Once & changes over time, the key joints' transformations are updated and linearly interpolated onto the Gaussians to obtain the final deformed positions and rotations, while other attributes in the Gaussian remain unchanged. Crucially, it is highly challenging to accurately optimize the SMPL poses (t) from scratch just with sensor observations, even for single-person or indoor scenarios (Jiang et al., 2022; Lei et al., 2023; Kocabas et al., 2024). Hence rough initialization of (t) is typically needed, whose details are deferred to a dedicated section $ 4.2.
Non-Rigid Deformable Nodes. These nodes act as a unified representation for other significant non-rigid instances, including those that fall beyond the scope of SMPL modeling, such as extremely faraway pedestrians for which even state-of-the-art SMPL estimators cannot provide accurate estima- tions, or ditto--distribution, template-less non-rigid individuals. Hence, we propose to use general deformation network Fsp with parameter yo to fit the non-rigid motions within the nodes. Specifically, for node h, the world-space Gaussians are defined as:
go (t) = Ta (L) s (g) <) Fo ee.. 4))
where the deformation network generates the changes of the Gaussian attributes from time f to the canonical space Gaussians Ødeform, outputting the changes in position 8jtn rotation Squ (t), and the scaling factors 6s, (t). The changes are applied back to g% @oedorm with the @ operator that internally performs simple arithmetic addition that results in (o, fe + 5p((), + 5q(t), s + 6s(t), c). Notably, previous approaches such as Yang et al. (2023c) utilizes a single deformation network for the entire scene, and usually fail in highly complex outdoor dynamic scenes with rapid movements. On the contrary, in our work, we define per-node deformation field which has much more representation power. To maintain computational efficiency, the network weights & are shared and the identities of the nodes are disambiguated via an instance embedding parameter en. Experimental results in $ 5.2 show that deformable Gaussians are essential for achieving good reconstruction quality.
Sky Node. We use a separate optimizable environmental map to fit the sky color from view- ing directions. Compositing the sky image Cay with the rendered Gaussians Cg consisting of (ghr ( }, (gSMPL)), ), (gh we obtain the final rendering as:
C = Cg + (1 Og)Cay:
(4)
where O₂ EE oi (1 - az)) is the rendered opacity mask of Gaussians.
4.2 RECONSTRUCTING IN-THE- WILD HUMANS
Reconstructing humans from driving logs faces challenges as in-the-wild pose estimators (Goel et al., 2023; Rajasegaran et al., 2022) are typically designed for single video input and often miss predictions in occlusion cases. We designed a pipeline that addresses these limitations to predict accurate and temporally consistent human poses from multi-view videos with frequent occlusions.
Formally, given a set of 3D tracklets for N pedestrians T = (T - from the dataset, our goal is to obtain the corresponding SMPL. pose sets: 0 = O h== Here, T,, and f (For brevity, (t) omitted) represent the boxes sequence and body pose sequence of the h-th human. We apply 4D-Humans (Goel et al., 2023) to each camera's video independently in our multi-camera setup. This yields separately processed results of human tracklets and poses: T = U2-1 The and 0 - 0c, where T = hepe and 0= = (Ø= Jiee= represent the predicted tracklets and poses from camera c, respectively. Here, Dr is the set of detected human indices in camera c. Our task is to reconstruct using& θ. We achieve this through the following steps:

---

Published as a conference paper at ICLR 2025
Traidet
Body Pose Prodetor 10: T
Inconsistent ID actoss Cameras
Consistent actoss Cameras
(a) Human ID Matching
(b) Pose Completion
Figure 3: Human Pose Processing. (a) Human ID matching ensures consistent identification across cameras. (b) Missing pose completion to recover poses of occluded individuals.
Tracklet Matching: We define a matching function M that finds the most similar predicted tracklets for each ground truth tracklet by computing the maximum mean IoU of their 2D projections:
On M(,, 0, T, T).
(S)
This function learns matching between ground truth tracklets and predicted tracklets, then outputs the corresponding matched pose sequences. Consider 3-camera setup as an example (Fig. 3(a)), if the h-th ground truth tracklet matches with detected tracklets jo, J1. 22 in cameras 0 to 2 respectively, then 0, (6% 0) 0 }. where 0% is the pose sequence from camera for the detected tracklet jk-
Pose Completion: As visualized in Fig. 3(b), 4D-Humans (Goel et al., 2023) fails to predict SMPL poses for occluded individuals in driving scenes, design process recover missing poses:
, H(@r.. T, T).
(6)
Here, function H. identifies missing detections by comparing the ground truth and predicted tracklets, and interpolates missing poses to complete 0g from On.
4.3 OPTIMIZATION
We simultaneously optimize all the parameters as mentioned in $ 4.1 a single stage to reconstruct the entire scene. These parameters include: (1) all the Gaussian attributes (opacity, mean positions, scaling, rotation, and appearance) their local spaces, namely ghv {Greid) {g (GSMPL.). ( (2) the poses of both rigid and non-rigid nodes for each frame I, i.e., (T (1)) A (1)). (3) the human poses of all the SMPI. nodes for each frame f, i.e., (0(t)) and the corresponding skinning weights, (4) the weight oo of the deformation network F. (5) the weight of the sky model.
We use the following objective function for optimization:
L L= X,) L1 + Ay + Adephh + Aapacity Capacity + Lreeg
where L1 and LSSIM are the LI and SSIM losses on rendered images, Luepth compares the rendered depth of Gaussians with sparse depth signals from LiDAR, Lupucity encourages the opacity of the Gaussians to align with the non-sky mask, and Lre represents various regularization terms applied to different Gaussian representations. Detailed descriptions of loss terms are provided the Appendix.
EXPERIMENTS
Dataset. We conduct experiments on the Waymo Open Dataset (Sun et al., 2020), which comprises real-world driving logs. We tested up to 32 dynamic scenes in Waymo, including eight highly complex dynamic scenes that, in addition to typical vehicles, also contain diverse dynamic classes such as pedestrians and cyclists. Each selected segment contains approximately 150 frames. The segment IDs are listed in Tab. 12 and Tab. 6. To further demonstrate our effectiveness on common driving scenes, extend our results to 5 additional popular driving datasets: NuScenes (Caesar et al., 2020), 2rgoverse2 (Wilson et al., 2023), PandaSet (Xiao et al., 2021), KITTI (Geiger et al., 2012), and NuPlan (Caesar et al., 2021).
Baselines. We compare our method against several Gaussian Splatting approaches: 3DGS (Kerbl al., 2023), DeformableGs (Yang al., 2023c), StreetGS (Yan et al., 2024), HUGS (Zhou et al.,
7

---

Published as a conference paper at ICLR 2025
Table I: Comparison on Waymo Open Dataset. We compute PSNR and SSIM for both the full image and dynamic regions. Vehicle indicates regions corresponding vehicle-related classes, while Human indicates regions corresponding to human-related classes. Box indicates methods that utilize bounding boxes for dynamic modeling. LiDAR means method using LiDAR information.
Rermedaattion
Vicw Sprthees
SSIM1
SSIM
013012
22:35
0.313
15.15
00:25
25.07
11385
20:32
11:55
25111
0.507
3DGS0Kcabl ct al, 2023)
26.00
0.912
16.85
0.414
15.15
0.425
25.57
0.925
16.62
0.387
16.00
0.407
Detomoi8tting 20233
2X.40
0.925
17:90
0.4(1
19.53
0.571
27.72
01922
17:30
0.426
19:91
0.530
#VaiChee et 2023)
38.37
0.927
24.05
0.503
25.02
0.787
30.19
0.919
21.30
0.567
22.25
0.579
HUns//hos st 2004)
0.484
34.31
0.794
0.94
15.09
0.328
0.764
StretOS: 2024)
0.420
27.73
25.5%
0.25
0.393
0.846
Ous
Ground Truth
(b) Ours
(c) EmerNeRF
(d) StreetGS
(c) HUGS
PVG
Figure 4: Qualitative Comparison of Novel View Synthesis. The insets highlight the details of the reconstructed dynamic objects. Omn i Re manages to recover very fine details, achieving high-quality reconstruction of various common dynamic objects, including vehicles, pedestrians, and cyclists.
2024), and PVG (Chen al., 2023). Additionally, we compare approach EmerNeRF (Yang et al., 2023a). Among methods compared, for StreetGS (Yan et al., 2024), we use our own reimplementation. For 3DGS (Kerbl al., 2023) and DeformableGS (Yang al., 2023c), we use the implementation with LiDAR supervision to ensure the comparison fairness. For other methods, we use their official code. For training, we utilize data from the three front-facing cameras, resized to resolution of 640x for all methods, along with LiDAR data for supervision. We utilize the instance bounding boxes provided by the dataset to transform objects and refine them via pose optimization during training. For further implementation details, please refer to Appendix.
method with NeRF-based
5.1 MAIN RESULTS
Appearance. We evaluate our method on scene reconstruction and novel view synthesis (NVS) tasks, using every 10th frame as the held-out set for NVS. We report PSNR and SSIM scores for full images, as well as human-related and vehicle-related regions, to assess dynamic reconstruction capabilities. The quantitative results in Tab. show that imniRe outperforms all other methods, with a significant margin in human-related regions, validating our holistic modeling of dynamic actors. Additionally, while StreetGS (Yan et al., 2024) and our method model vehicles in a similar way, observe that OmniRe is slightly better than StreetGS even in vehicle regions. This is due to the absence human modeling StreetGS, which allows supervision signals from human regions (e.g., colors, LiDAR depth) incorrectly influence vehicle modeling. The issues StreetGS faces one of our motivations for modeling almost everything in a scene holistically, aiming to eliminate erroneous supervision and unintended gradient propagation.
In addition, we show visualizations in Fig. to assess model performance qualitatively. Although PVG (Chen et al., 2023) performs well on the scene reconstruction task, it struggles with the
8

---

Published as a conference paper at ICLR 2025
novel view synthesis task in highly dynamic scenes, resulting in blurry dynamic objects in novel views (Fig. 4-(D). HUGS (Zhou et al., 2024) (Fig. 4-(e)), StreetGS (Yan et al., 2024) 4-(d)) and 3DGS (Kerbl et al., 2023) (Fig. 8-(h)) fail to recover the pedestrians because they not capable of modeling non-rigid objects. DeformableGS (Yang et al., 2023c) (Fig. 8-(g)) suffers from extreme motion blur for outdoor dynamic scenes with rapid movements, despite achieving reasonable performance for indoor scenes and cases with small motion. EmerNeRF (Yang et al., 2023a) reconstructs coarse structures of moving humans and vehicles certain level, but struggles with fine-grained details(Fig. 4-(c)). In contrast to all these methods in comparison, our method faithfully reconstructs fine details for any object the scene, handling occlusion, deformation, and extreme motion. Video comparisons are included in the project page.
Geometry. In addition to appearance, we also investigate whether our method can reconstruct fine geometry of urban scenes. We evaluate the Root Mean Squared Error (RMSE) and two-way Chamfer Distances (CD) for LiDAR depth reconstruction on both training frames and novel frames. Details about evaluation procedures are provided in the Appendix. Tab. 3 reports the results. Our method outperforms others by a large margin. Fig. 5 illlssrates the accurate reconstruction of dynamic actors achieved by method in comparison other approaches.
(a) GT
(b) Ours
(c) StreetGS
(d) DeformGS
(e) 3DGS
Figure 5: Visualizations of Rendered LiDAR. Our method accurately reconstructs LiDAR data for humans and vehicles compared to other approaches.
5.2 ABLATION STUDIES & APPLICATIONS
SMPL Modeling, SMPL modeling is important to model the local, continuous movements of humans. We study its impact by disabling the human pose transformation enabled by SMPL and report the results in Tab. 2 ((a) v.s. (b)) and illustrate these effects in Fig. 7-(B). Without template-basee modeling, the reconstructed human renderings appear highly blurred, particularly around the legs, thus failing to accurately reconstruct human body movements. This contrasts sharply with the precise leg reconstruction observed our default setting. Morcover, SMPL modeling provides joint-level control, improving the controllability (Fig. 1-(c.3), (c,4)).
The human body poses extracted as described in ($ 4.2) exhibit prediction emrrs and scale ambiguity, which subsequently lead to pose errors that degrade reconstruc- tion quality, as shown in Fig. 6 (Noisy). We improve this by jointly optimizing the human poses and Gaussians via the showcases the refined poses. These results verify the effectiveness of our refinement strategy.
Human Body Pose Refinement.
reconstruction losses. Tab. 2-(a) (c) ablates this design choice, and Fig. 6
Deformable Nodes. Deformable nodes are important for accurately reconstructing aut-of-distribution or template-less actors. Our approach addresses this challenge by learning a self-supervised deforma- tion field that transforms Gaussians from their canonical space the shape space. Tab. 2 ((a) v.s. (d) proves the importance of this component. In Fig. 7-(A) shows that without deformable nodes, some dynamic actors are either ignored or incorrectly blended into the background.
Boxes Refinement. In practice, we observe that the instance bounding boxes provided by the dataset are imprecise. These noisy ground truth boxes can be harmful to rendering quality. To address this, we jointly refine the bounding box parameters during training. Tab. 4 and Fig. 12 show the practical
Table 2: Ablation on Ron-Rigid Modeling.
Table 3: Evaluation of LiDAR Depth Accuracy.
Full PSNR
Human PSNR
Traiiig Frames
Recon.
NVS
Recon.
NVS
Methads
CDC
RMSEL
CD)
RMSEJ
(a) Ours default
34,25
32,57
28.15
24.36
3DGS/Kedt al, 2023)
0.415
2,804
01467
2,396
SMPL
32.50
31.76
DeformableGS// 2023c)
0.384
2.965
2.950
(c) wlo Body pose refine
33.84
32.44
26.97
24.04
StreeCS(Yaa al., 2024)
0.274
(d) w/o Deformed actors
33.64
32.17
25.26
22.41
0.242
1.894
0.244

---

Published as a conference paper at ICLR 2025
Noisy
Refined
Noisy
Refined
Noisy
Refined
Noisy
Refined
Figure 6: Ablation of Human Body Pose Refinement.
w/o Deformed Actors
Deformed Actors
w/o SMPL
w/ SMPL
Figure 7: Ablation of Human Modeling.
benefits of this simple yet effective step, which results in improved numeric metrics and reduced blurriness of foreground objects.
Applications to Simulation. Thanks to the decomposition nature of Omn Re, each instance is modeled separately. After joint training, we obtain reconstructed assets that can be flexibly edited in terms of position and rotation. Beyond editing within single scene, we can also transfer assets from one scene to another, adding variety and complexity to the reconstructed environments. Fig. (cc,left) demonstrates swap of the black vehicle originally in the scene (inset) with a reconstructed vehicle from another scene; and (c,right) an insertion of pedestrian from the scene in the inset to Fig. 11. Through explicit modeling of pedestrians and other non-rigid individuals, we achieve the simulation of reenacted scenarios involving detailed pedestrian-vehicle interaction. As demonstrated in Fig. 9, we simulate a moving vehicle stopping at a crossing, waiting for a pedestrian who slowly crosses. The pedestrian is reconstructed from another scene. This simulation of humans is extremely challenging for previous methods. This level of precise control over reconstructed photorealistic assets opens up possibilities for integration with previous simulators for automated simulation (Wang et al., 2022; Wei et al., 2024)
Table 4: Ablation on GT Boxes Refinement.
Full PSNR
Human PSNR
Vehicle PSNR
Recon.
NVS
Recon.
NVS
Recon.
NVS
Complete Model
34,25
32,57
28.15
24.36
28.91
27.57
w/o Box Refine.
33.04
31.72
26.53
23.67
25.57
24.78
scene. Additional
swap edits shown in
6 CONCLUSION
Our method, OmniRe, tackles comprehensive urban scene modeling using Gaussian Scene Graphs. It achieves fast, high-quality reconstruction and rendering, suggesting promise for driving and robotics simulation. We also present solutions for human modeling in complex environments. Future work includes self-supervised learning, improved scene representations, and safety/privacy considerations. To ensure reproducibility, the code is available at link.
Broader impact. Our method aims to address a significant problem in autonomous driving- simulation. This approach has the potential enhance the development and testing of autonomous vehicles, potentially leading to safer and more efficient AV systems. Simulation, in safe and controllable manner, remains an open and challenging research question.
Limitations. While enabling holistic scene modeling, CmniRe still has certain limitations. First, our method does not explicitly model lighting effects, which may lead visual harmony issues during simulations, particularly when combining elements reconstructed under varying lighting conditions. Addressing this non-trivial challenge requires dedicated efforts beyond the scope of our current work. Further research into modeling light effects and enhancing simulation realism remains crucial for achieving more convincing and harmonious results. Second, similar to other per-scene optimization methods, Omn MnniRe produces less satisfactory novel views when the camera deviates significantly from the training trajectories. Future works to address this issue include incorporating data-driven priors, such image or video generative models, and optimizing camera poses jointly.
10

---

Published as a conference paper at ICLR 2025
7 ETHICS STATEMENT
Our work does not involve the collection or annotation of new data. We utilize well-cstablished public datasets that adhere to strict ethical guidelines. These datasets ensure that sensitive information, including identifiable human features, blurred or anonymized to protect individual privacy. We are committed to ensuring that method, as well as future applications, employed responsibly and ethically to maintain safety and preserve privacy.
8 ACKNOWLEEDGMMENT
This work supported by funding from Toyota Research Institute, Dolby, and Google DeepMind. Yue Wang is also supported by a Powell Faculty Research Award. We also thank Jiageng Mao, Junjie Ye, Ziyi Yang, Haozhe Lou, and Yifan for their valuable discussions during the project, which helped us resolve issues and improve the methods.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, llge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 0tt::3333.087744 2023.
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5855-5864, 2021.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5470-5479, 2022.
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11621-11631, 2020.
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, and Sammy Omari. nuplan: A closed-loop ml-based planning benchmark for autonomous vehicles. arXiv preprint arXiv: 106. /180, 2021.
Hongrui Cai, Wanquan Feng, Xuetao Feng, Yan Wang, and Juyong Zhang. Neural surface recon- struction of dynamic scenes with monocular rgb-d camera. In Thirty-sixth Conference on Neural Information Processing Systems (NeurIPS), 2022.
Yurui Chen, Chun Gu Junzhe Jiang, Xiatian Zhu, and Li Zhang. Periodic vibration gaussian: Dynamic urban scene reconstruction and real-time rendering. arXiv preprint arXiv:2311.18561. 2023.
Tobias Fischer, Jonas Kulhanek, Samuel Rota Bulò, Lorenzo Porzi, Marc Pollefeys, and Peter Kontschieder. Dynamic 3d gaussian fields for urban areas. arXiv preprint arXiv:2406.0317. 2024a.
Tobias Fischer, Lorenzo Porzi, Samuel Rota Bulo, Mare Pollefeys, and Peter Kontschieder. Multi- level neural scene graphs for dynamic urban environments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21 225-21113 2024b.
Sara Fridovich-Keil. Alex Yu. Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5501-5510 2022.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 2012 IEEE conference on computer vision and pattern recognition, pp. 3354-3361. IEEF, 2012.
11

---

Published as a conference paper at ICLR 2025
Shubham Goel, Georgios Pavlakos, Jathushan Rajasegaran, Angjoo Kanazawa, and Jitendra Malik. Humans in 4D: Reconstructing and tracking humans with transformers. In ICCV, 2023.
Jianfei Guo, Nianchen Deng, Xinyang Li, Yeqi Bai, Botian Shi, Chiyu Wang, Chenjing Ding, Dongliang Wang, and Yikang Li. Streetsurf: Extending multi-view implicit surface reconstruction to street views. arXiv preprini arXiv: 2tli:2330004888. 2023.
Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. arXiv preprint arXiv:, 2023.
Yi-Hua Huang, Yang- Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Wei Jiang. Kwang Moo Yi, Golmoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In European Conference on Computer Vision, pp. 402-418. Springer, 2022.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimküüler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions Graphics, 42(4), July 2023. URL https: http://eeppeee//e sam. anria.fr fungraph/ banggraph/dd-gaaaee aodd-guussin an- spwattinigin
Muhammed Kocabas, Jen-Hao Rick Chang, James Gabriel, Oncel Tuzel, and Anurag Ranjan. HUGS: Human gaussian splatting. In 2024 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024. URL httptttt///// http///axxvv.rrr/
Abhijit Kundu, Kyle Genova, Xiaoqi Yin, Alireza Fathi, Caroline Pantofaru, Leonidas J Guibas, Andrea Tagliasacchi, Frank Dellaert, and Thomas Funkhouser. Panoptic neural fields: seman- tic object-aware neural scene representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12871-12881, 2022.
Jiahui Lei, Yufu Wang, Georgios Pavlakos, Lingjie Liu, and Kostas Daniilidis. Gart: Gaussian articulated template models. arXiv preprint 1.X 2023.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, and Zhaoyang Lx. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2022.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black. SMPL: A skinned multi-person linear model. volume 34, pp. 248:1-248:16 16. ACM, October 2015.
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024.
Ricardo Bratala, Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Dosovitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7210-7219 2021.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Taneik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. In ECCV, 2020.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (TOG), 41(4): 1-15, 2022.
Julian Ost. Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide. Neural scene graphs for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2856-2865. 2021.
12

---

Published as a conference paper at ICLR 2025
Keunhong Park, Utkarsh Sinha, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Steven M. Seitz, and Ricardo Martin- Martin-Bruulla, Nerfies: Deformable neural radiance fields. ICCV, 2021a.
Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T. Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M. Seitz. Hypernerf: A higher-dimensionall representation for topologically varying neural radiance fields. ACM Trans. Graph, 40(6), dee 2021b.
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-NeRF: Neural Radiance Fields for Dynamic Scenes. Proceedings of the IEEE/CVE Conference on Computer Vision and Puttern Recognition, 2020.
Jathushan Rajasegaran, Georgios Pavlakos, Angjoo Kanazawa, and Jitendra Malik, Tracking people by predicting 3D appearance, location & pose. In CVPR, 2022.
Konstantinos Rematas, Andrew Liu, Pratul P Srinivasan, Jonathan T Barron, Andrea Tagliasacchi. Thomas Funkhouser, and Vittorio Ferrari. Urban radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12932-12942, 2022.
Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2446-2454, 2020.
Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan Barron, and Henrik Kretzschmar, Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8248-8258, 2022.
Adam Tonderski, Carl Lindström, Georg Hess, William Ljuugherghh Lennart Svensson, and Christof- fer Petersson. Neurad: Neural rendering for autonomous driving. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14895-14904, 2024.
Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In IEEE International Conference on Computer Vision (ICCV). IEEE, 2021.
Haithem Turki, Jason Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 02375-12385 2023.
Cheng Yao Wang, Oron Nir, Sai Vemprala, Ashish Kapoor, Eyal Ofek, Daniel McDuff, and Mar Gonzalez-Franco. Citylifesim: A high-fidelity pedestrian and vehicle simulation with complex behaviors. In 2022 IEEE 2nd International Conference on Intelligent Reality (ICIR), pp. 11-16. IEEE, 2022.
Jingbo Wang, Ye Yuan, Zhengyi Luo, Kevin Xie, Dahua Lin, Umar Iqbal, Sanja Fidler, and Sameh Khamis. Learning human dynamics in autonomous driving scenarios. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 20796-20806, 2023.
Jingbo Wang, Zhengyi Luo, Ye Yuan, Yixuan Li, and Bo Dai. Pacer+: On-demand pedestrian anima- tion controller in driving scenarios. In Conference Computer Vision and Pattern Recognition (CVPR), 2024.
Yuxi Wei, Zi Wang, Yifan Lu, Chenxin Xu, Changxing Liu, Hao Zhao, Siheng Chen, and Yanfeng Wang. Editable scene simulation for autonomous driving via collaborative I1m- llm-agents. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). June 2024.
Benjamin Wilson, William Qi, Tanmay Agarwal, John Lambert, Jagjeet Singh, Siddhesh Khandelwal, Bowen Pan, Ratnesh Kumar, Andrew Hartnett, Jhony Kaesemodel Pontes, et al. Argoverse 2: Next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00499, 2023.
13

---

Published as a conference paper at ICLR 2025
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint 230.0:2310..8528, 2023a.
Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D² Self-supervised decoupling of dynamic and static objects from a monocular video. Advances in Neural Information Processing Systems, 2022.
Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang, Yuxin Huang, Xiaoyu Ye, Zike Yan, Yongliang Shi, Yiyi Liao, and Hao Zhao. Mars: An instance-aware, modular and realistic simulator for autonomous driving. CICAL, 2023b.
Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9421-9431, 2021.
Pengchuan Xiao, Zhenlei Shao, Steven Hao, Zishuo Zhang, Xiaolin Chai, Judy Jiao, Zesong Li, Jian Wu, Kai Sun, Kun Jiang, et al. Pandaset: Advanced sensor suite dataset for autonomous driving. In 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 3095-3101. IEEE, 2021.
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. In Neural Information Processing Systems (NeurIPS), 2021.
Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint arXiv:2401.01333 2024.
Jiawei Yang. Boris Ivanovic, Or Litany, Xinshuo Weng. Scung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, and Yue Wang. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. In International Conference on Learning Representations, 2023a.
Ze Yang, Sivabalan Manivasagam, Ming Liang, Bin Yang. Wei-Chiu Ma, and Raquel Urtasun. Recovering and simulating pedestrians in the wild. In Conference on Robot Learning, pp. 419-431. PMLR, 2021.
Ze Yang, Yun Chen, Jingkang Wang, Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1389-1399, 2023b.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formahle 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309. 13101, 2023c.
Zongxin Ye, Wenyu Li, Sidun Liu, Peng Qiao, and Yong Dou. Absgs: Recovering fine details for 3d gaussian splatting. arXiv preprint arXiv:2404. 10444, 2024.
Hongyu Zhou, Jiahao Shao, Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. arXiv preprint arXiv:2403.1277 2024.
Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming- Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arXiv:2312.07920, 2023.
14

---

Published as a conference paper at ICLR 2025
Supplemental Material
A IMPLEMENTATION DETAILS
Initialization: For the background model, we refer to PVG (Chen et al., 2023), combining 6 x 10 LiDAR points with x 10 random samples, which are divided into 10y near samples uniformly distributed by distance to the scene's origin and 2 x 10 far samples uniformly distributed by inverse distance. To initialize the background, we filter out the LiDAR samples of dynamic objects. For rigid nodes and non-rigid deformable nodes, utilize their bounding boxes to accumulate the LiDAR points, while for non-rigid SMPL nodes, we initialize the Gaussians on the template mesh in their canonical space. To determine the initial color of Gaussians, we project LiDAR points onto the image plane, whereas random samples are initialized with random colors. The initial human body pose sequences of non-rigid SMPL Nodes are obtained through the process described in 4.2.
Our method trains for 30,000 iterations with all scene nodes optimized jointly. The learning rate for Gaussian properties aligns with the default settings of 3DGS (Kerbl al., 2023), but varies slightly across different node types. Specifically, we set the learning rate for the rotation 10 for non-rigid SMPL nodes and 10 for other nodes. The degrees of spherical harmonics are set to 3 for background nodes, rigid nodes, and non-rigid deformable nodes, while it set to for non-rigid SMPL nodes. The learning rate for the rotation of instance boxes is 1 x 10-5, decreasing exponentially to 5 x 10 The learning rate for the translation of instance boxes is x 10-4, decreasing exponentially to 10-4, The learning rate for human body poses of non-rigid SMPL nodes is 5 x 10-5, decreasing exponentially to 1 x 10 10-5, For the Gaussian densification strategy, we utilize the absolute gradient of Gaussians introduced in Ye et al. (2024) to control memory usage. We set the densification threshold of position gradient to x 10 10-4. This use of absolute gradient has a minimal impact on performance, as discussed in detail in Appendix D.4. The densification threshold for scaling is x 10-3 Our method runs single NVIDIA RTX 4090 GPU, with training for each scene taking about hour. Training time varies with different training settings.
Training:
of Gaussians
Optimization: We utilize the loss function introduced in Eq (7) to jointly optimize all learnable parameters. The image loss is computed as:
Limage A, CSSIM
(8)
due to sparse comporal-spatian observation of the dynamic part, its supervision signal is insufficient. To address this, we apply higher image loss weight to the dynamic regions identified by the rendered dynamic mask. This weight is set 5. The depth map loss is computed as:
how Σ E|
-b1,
Leeeth
where D is the inverse of the sparse depth map. We project LiDAR points onto the image plane to generate the sparse LiDAR map, and D is the inverse of the predicted depth map.
The mask loss Lapacity is computed as:
Lopuiity how Σog log Og
1
1
E log(1 - Og)
(10)
lew
where May is the sky mask, and Og is the rendered opacity map.
In addition to the reconstruction losses, we introduce various regularization terms for different Gaussian representations to improve quality, Among these, an important regularization term Cpose designed to ensure smooth human body poses (t). This term is defined as:
Epese
0 6) + O(t + 5) 20(
(11)
2
where 5 is randomly chosen integer from {1, 2, 5). We set the weight of the SSIM loss, X,. to 0.2, the depth loss, Adepth. to 0. the opacity loss, Xuqucityy to 0.05, and the pose smoothness loss, Apue, 0.01.
15

---

Published as a conference paper at ICLR 2025
(g) DeformableGS
(h) 3DGS
Figure 8: Additional Qualitative Comparison of Novel View Synthesis.
Figure 9: A sample of human-vehicle interaction simulation in driving scenarios.
B BASELINES
EmerNeRF (Yang et al., 2023a) is a state-of-the-art NeRF-Based method for dynamic driving scene reconstruction. EmerNeRF uses static field represented by a 3D Hash-Grid model the static parts of the scene and a dynamic field with 4D Hash-Grid to model the dynamic parts. Additionally, it employs flow field to aggregate the dynamic features. This self-supervised decomposition approach yields good results on dynamic scene modeling and static-dynamic decomposition, with the scene flow emerging in the flow field.
DeformableGS (Yang et al., 2023c) defines a canonical space to represent scenes with Gaussians. To model dynamics, uses deformation network to predict offsets of Gaussian properties. These offsets then deform the Gaussians to fit the scene dynamics. DeformableGS works well in synthetic and indoor datasets. We compare to our method to evaluate its ability to model challenging out-door dynamic scenes.
StreetGS (Yan et al., 2024) is a dynamic scene modeling method based on Gaussian Splatting for driving scenes. StreetGS models the components of dynamic scenes separately: the static background and the foreground vehicles. It utilizes boxes predicted by an off-the-shelf model to warp the Gaussians of foreground vehicles and refine them during training. StreetGS yields good results driving scenes but ignores other non-rigid dynamic objects in the scene.
HUGS (Zhou al., 2024) is GS-based method for driving scene modeling and understanding. It not only models the appearance of a scene but also distills 2D flow maps and semantic maps into the 3D scene enable holistic urban
understanding. Similar StreetGaussian (Yan et al., 2024), HUGS uses object boxes for compositing dynamic elements. HUGS achieves good performance in both scene modeling and semantic modeling. However, it primarily focuses on rigid backgrounds and objects, without addressing non-rigid dynamics.
PVG (Chen al., 2023) introduces Periodic Vibration Gaussians that vibrate over time with optimizable vibration directions, life span, and life peak (the moment of highest opacity) to represent dynamic scenes. These Gaussians are optimized using self- supervipedd approach. The method achieves static-dynamic decomposition by categorizing Gaussians based on their life spans. We compare PVG with our method to evaluate our capability modeling highly complex dynamic scenes.
Among all the compared methods, HUGS (Zhou et al., 2024) and StreetGaussians (Yan et al., 2024) require bounding boxes of foreground objects. PVG (Chen et al., 2023), StreetGaussians (Yan et al., 2024), and EmerNeRF (Yang et al., 2023a) utilize LiDAR data for depth supervision. While the original implementations of 3DGS (Kerbl et al., 2023) and DeformableGs (Yang al., 2023c) do not include depth supervision, we added LiDAR depth supervision in experiments to ensure fair comparison.
16

---

Published as a conference paper at ICLR 2025
C EVALUATION
Appearance. For the Novel View Synthesis task, select every 10th frame from the original sequence as the test set. We use PSNR and SSIM to evaluate the quality of the rendered images. also compute the PSNR and SSIM for regions with vehicles
Since focus dynamic scenes, and humans. To identify regions of vehicles and humans, we use Segformer (Xie et al., 2021) to obtain semantic masks. We further identify the movable dynamic parts using projections of moving object bounding boxes, utilizing their velocity information. One example of dynamic masks cam bee seen in Fig. 10.
GT Image
Human Region
Vehicle Region
Figure 10: An example the dynamic masks for computing dynamic region metrics.
Geometry. Our method uses LiDAR data initialize Gaussians and supervise scene depth by comparing the rendered depth map with the sparse LiDAR depth map. Post-training, Gaussians typically deviate from their initial state through densification optimization. Therefore, comparing the LiDAR depth reconstruction is still a valid comparison. We follow the depth evaluation method of StreetSurf (Guo et al., 2023): render a depth map and match depth pixels LiDAR rays. For Chamfer Distance, re-project the predicted depth to 3D using the LiDAR ray direction and origin. For RMSE, compare the GT and predicted ranges for LiDAR rays.
D ADDITIONAL RESULTS
D.1 QUALITATIVE COMPARISON
We recommend readers our project page for video comparisons the methods.
D.2 QUANTITATIVE COMPARISON
To further validate our method's s effectiveness, we tested our method against StreetGS (Yan et al., 2024) and EmerNeRF (Yang et al., 2023a) on 32 dynamic scenes from the Waymo dataset, with results reported in Tab. 5.
Table 5: We expanded our evaluation to 32 dynamic scenes from the Waymo dataset, comparing our method with StreetGS (Yan et al., 2024) and EmerNeRF (Yang et al., 2023a). The segment IDs listed Tab. 6.
Scene Reconstruction
Novel View Synthesis
Full Image
Human
Vahicle
Human
Vehicle
Methods
PSNRS
SSIM
PSNR"
SSIMI
PSNRT
SSIMI
PSNKT
SSIMI
PSNR1
SSIMI
PSNRT
SSIM
EmezNeRI
31.29
0.877
23.14
0.551
21.17
0.700
29.04
0.311
20.76
0.167
21.50
0.582
Streee[SS
29.93
0.931
19.63
0.524
27.48
0.871
28.73
0.910
18.77
0.470
26.18
0.825
Ours
33.33
0,800
31.1
24.57
Table 6: Segment IDs of 32 dynamic scenes of Waymo Dataset used in the test for Tab. 5.
Neg 1015555....
25030...
44..
1eg584622
seg776165..
138251...
s189418777...
seg9653244..
seg
seg 22514
aeg
segi 940144..
sog 166004
seg 173881
150215148.
Sog 164...
se5454855...
Ngg560223...
57 325..
seg5870666..
4xp842457
169052165...
1g0952995
12365...
Ngp 52664...
sgell 1445...
Agg 23218...
segl 02252...
sgg148106...
sgg656611.
segl 179334..
seg 104859...
17

---

Published as a conference paper at ICLR 2025
D.3 OMNIRE ON CHALLENGING SCENES
To demonstrate the effectiveness and robustness of OmniRe, we tested our method alongside StreetGS (Yan et al., 2024), PVG (Chen al., 2023), and DeformableGS (Yang et al., 2023c) on various challenging scenes. The results show that OmniRe maintains high reconstruction qual- ity under most challenging conditions. This section provides the quantitative performance of the compared methods. Video visualizations and comparison are included in our project page.
Super Crowded Scenes. Reconstructing scenes with extreme dynamic occlusions poses signifi- cant challenges. Omn Re specifically designed to handle these scenes and performs well even in highly occluded scenes. We conducted experiments on three extremely crowded scenes from Waymo (segl 12520.... seg 104859..... seg152664... including situations with large crowd of people simultaneously crossing street. Results in Tab. 7 show that OmniRe demonstrates strong performance in these scenes.
Table 7: Comparison super crowded scenes.
Soane Reconsrrccton
Novel View Synthesis
Full Image
Human
Velicle
Full Image
Human
Vehicle
Methods
PSNR1
SSIMT
PSNR
SSIM
PSNR
SSIMI
PSNR
SSIM
PSNR1
SSIMT
PSNR1
SSIM1
DetormGS
26.69
0.861
19,48
0.504
17,49
0.455
25.66
18.71
0.448
17.12
0.14
PVG
29.11
0.865
24.56
0.686
21.29
0.644
27.21
0.830
22.24
0.555
19.63
0.528
StreetGS
26.81
0.965
19.31
0.516
24.79
0.787
25.67
0.841
18.53
0.459
23.40
0.723
Ours
27.50
0.809
25.91
0.812
28.91
23.78
24.46
0,750
Nighttime Scenes.
We tested three nighttime scenes Segg12900.... seg 1eg102261.... seg 128560...)
from Waymo. Results in Tab. 8 indicate that OmniRe outperforms the compared methods, achieving superior reconstruction quality under low-light conditions.
Table 8: Comparison night time scenes.
Scene Recomstracion
Synthesis
Poll Image
Human
Nchicle
Full Image
Human
Vahicle
Methods
PSNR1
SSIM1
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR1
SSIMI
PSNR1
SSIM1
DetormGS
20.62
PVG
30.55
0.768
30.74
0.777
22.77
0.713
29.32
0.740
25.22
0.614
20.90
0.611
StreaGSS
30.60
0.775
27.51
0.667
25.68
0.772
29.38
0.741
21.88
0.523
24.61
0.729
29.79
Adverse Weather Conditions. To evaluate performance under adverse weather conditions, we tested seven scenes with varying weather types: rainy (seg1 13555.... ... seg neg109277.... seg1413339...) foggy (seg 611022... leg172163... c. cloudy (seg144275. ... seg157956.. Omn Re handled these challenging conditions robustly, maintaining high reconstruction fidelity across all weather conditions (Tab. 9).
Table 9: Comparison adverse weather scenes.
Keconstruction
Novel Symthesis
Full Image
Human
Vehicle
Full Image
Human
Vchicle
Methods
PSNRT
SSIMt
PSNR
SSIM
PSNR
SSIM
PSNR1
SSIM?
PSNRT
SSIM
PSNRT
SSIM
DetormGS
23.12
31.45
PVG
32.75
0.927
27.75
0.785
27.20
0.795
31.12
0.907
25.02
0.656
24.44
0.677
StreetGS
33.49
0.933
18.26
0.424
32.28
0.916
32.05
0.918
18.56
0.434
29.60
0.841
0.857
32.55
0.919
32.58
0.844
High-Speed Scenes. For high-speed scenes such as highways, we tested three highway scenes from the Waymo dataset (seg 109239...... segl 13924.... segl (1391.396). Results are provided in Tab. 10. As highways typically lack non-rigid objects (e.g., humans), human metrics were not applicable. Since non-rigid modeling was not required in these scenes, OmniRe ie and StreetGS demonstrated comparable performance.
Table 10: Comparison high speed scenes.
Scene Reconsruction
Novel View Synthesis
Foli Image
Human
Vobacle
Full Image
Homan
Vahicle
Methods
PSNR1
SSIMI
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM?
PSNR1 SSIMI
PSNRT
SSIM1
DeformGs
0.855
17.49
0.482
25.56
0.407
PVG
25.28
0.861
21.14
0.617
25.79
0.825
17.65
0-170
StreesGS
30,89
0.902
29.45
0.883
28.05
0.863
24,88
0.766
29.51
0.760
18

---

Published as a conference paper at ICLR 2025
D.4 ABLATION STUDIES
Absolute Gradient. In our implementation, we applied AbsGrad for 3DGS densification across all reproduced methods (StreetGS, DeformableGS, and 3DGS) as a standard practice. To quantity the impact of AbsGrad, results of this analysis are presented in Tab. 11. We see that disabling AbsGrad leads to a marginal performance decrease (about 0.1 PSNR) for all methods, proving that AbsGrad is not the key factor contributing to our performance advantage over others, Note that DeformableGS fails to run due to out-of-memory issues when AbsGrad disabled. Based on these findings, we recommend the incorporation of AbsGrad as a standard practice in 3DGS densification and related methods.
conducted comparative study with and without its application. The
Table 11: Ablation on AbsGrad. By default, we apply AbsGrad to all GS-based approaches
reproduced by We now disable it to analyze its impact. We mark methods with AbsGrad enabled with grey background. We observe that 1) DeformableGS fails under w/o. AbsGrad setting because of out of memory issue; 2) enabling AbsGrad is good practice (+0.1 PNSR for all methods) but not an enabling factor for our performance lead.
Scene Reconstruction
Novel View Synthesis
Image
Vehicle
Image
Human
Vchicle
Mothods
PSNR1
SSIM
PSNR
SSIMI
PSNR1
SSIM
PSNR
SSIM1
PSNR1
SSIM
PSNR
SSIMI
EmerNeRT
31.95
0.902
22,88
0.378
24.65
0.723
29.67
0,383
10.32
0.454
22,07
0.609
3DG
D912
16.18
0.906
16.62
0,407
3DGS
25.84
0910
16.69
16,02
25.61
0.905
16.52
0.405
DetormGg
28.40
0.929
17.80
0,460
19.53
0.570
27.72
0.922
17.30
0.426
18.91
0,330
DeformGS
0.937
21.30
0.679
HUGS
28.26
0.923
16.23
0.404
24.31
0.794
27.65
0.914
15.99
0.378
23.27
0.748
St
29.08
0.936
16.83
0.420
27.73
0.880
28.54
0.9928
16:55
01:99
26.71
0.846
StreetGS
28.89
0.932
16.70
0.409
28.07
0.878
28.46
0.926
16.41
0.387
26.86
0.845
Our
34.25
0.954
28.15
0.845
28.91
0.892
32.57
0.942
24.36
0.727
27.57
0.858
Ours
34.11
0.953
28.00
0.842
28.83
0.890
32.16
0.941
24.28
0.726
27.55
0.857
Table 12: Segment IDs of 8 dynamic scenes of Waymo Dataset used in the test for Tab. 1. and Tab. 11
seg 10150454...
125025000...
1eg169514...
Seg584622... s0g776165...
Negl 38251...
80y4487677..
$29965324
Additional Results The Tab. 14 is the full table of Tab. that includes evaluation on SSIM. The Tab. 13 is the full table of Tab. 4 that includes evaluation SSIM.
Table 13: Ablation GT Boxes Refinement.
Reconstruction
Synthesis
Pall Image
Human
Vehicle
Full Image
Humaa
Vehicle
PSNR1 34.25
SSIM
PSNR1
SSIM
PSNR1
SSIM
PSNR
SSIM
PSNR
SSIM
PSNR1
SSIM
Complete Model
0.954
28.15
0,845
28.91
0.892
32,57
0.442
24.36
0.727
27.57
0.858
Box Refine.
0.947
0.790
31.72
23.67
24.78
E OMNIRE IN PRACTICE
Bounding Boxes. Similar to other scene-graph-based approaches (Ost et al., 2021; Yang et al.,
2023b; Tonderski al., 2024; Fischer al., 2024b; Zhou et al., 2023; 2024; Yan al., 2024), utilize bounding boxes for driving scene reconstruction, as they are widely used for producing superior reconstruction results compared methods that do not employ them. Additionally, bounding boxes offer significant controllability. It allows precise manipulation of both rigid objects like vehicles and non-rigid objects such as individual human body movements an ability lacking in self-supervised methods like EmerNeRF (Yang et al., 2023a) and PVG (Chen et al., 2023) that do not use instance information. This level controllability is crucial for tasks like scene simulation, which require the ability to manage the movement of all participating agents. Lastly, bounding box annotation is standard and generally straightforward process the autonomous driving field, with most popular datasets already providing these annotations via established auto-labeling tools, thereby minimizing manual effort and making the resource both efficient and accessible. For real world driving logs, these auto-labeling tools can generate precise bounding boxes at little cost.
19

---

Published as a conference paper at ICLR 2025
Table 14: Ablation on Non-Rigid Modeling.
Scenc Reconstruction
Novel View Synthesis
Full
Image
Human
Image
PSNR1
SSIM
PSNR1
SSIM
PSNR1
SSIM
PSNR1
SSIM
(a) Ours default
34,25
0.954
28.15
0.845
32.57
0.942
24.36
0.727
(b) SMPL actors
24.71
0.770
31,76
23.18
0.694
wo Body Pose Refine.
33.84
0.952
26.97
0.815
32.44
0.941
24.04
0.712
(d) w/o Deformed actors
33.64
0.953
25.26
0.766
32.17
0.941
22.41
0.653
Reconstructed Background
Original
Insert
Insert
Figure 11: Our method allows for flexible editing of scene assets.
How Determine Gaussian Representations for Humans? We categorize pedestrians into two groups for modeling. Near-range pedestrians, detected by our human pose processing module introduced in $ 4.2, are modeled using SMPL nodes. Far-range pedestrians, typically undetected due to distance, are modeled using deformable nodes. This approach naturally distinguishes between near and far-range pedestrians based on human detection capabilities.
Other individuals, such those using wheclchairs, skatcboards, or bicycles, often labeled as "cyclists" the datasets we study. However, these labels may be specific to the dataset used, and in some cases, the annotations might not be accurate. For instance, in the Waymo Dataset, person motorcycle may be labeled as a "vehicle". This reliance on dataset-specific labels could potentially limit the generalization of method other scenarios with imperfect labels.
To address this issue, we conducted preliminary experiments using GPT-4o (Achiam et al., 2023) to classify individuals (cropped bounding boxes) into categories: pedestrians and humans using personal transportation devices (e.g., wheelchairs, bicycles, motorcycles). Testing on 60 individuals (30 from each category), GPT-4o (Achiam al., 2023) achieved 100% accuracy. This suggests that accurate labels can be obtained relatively easily, thanks to the development of vision-language models.
Ground Truth
w/ Box Refine.
w/o Box Refine.
Figure 12: Ablation of Boxes Refinement.
20


Reviews:
[{"review_id": "62zsxzcw84", "reviewer": "Reviewer_WVni", "summary": "OmniRe is a framework designed to create high-fidelity digital twins of dynamic urban scenes for simulations, particularly for applications in autonomous driving. OmniRe goes beyond vehicle modeling to support diverse dynamic actors like pedestrians and cyclists, enabling complex simulations that reflect real-world scenarios. It utilizes Gaussian Scene Graphs with multiple representations, allowing detailed and editable scene reconstructions with both rigid (e.g., vehicles) and non-rigid (e.g., pedestrians) actors.", "questions": "I wonder for items like causality and new synthesis if an approach more configurable could take place now that they have separated the pedestrians from the road. \n\nThinking of something like this \nWang, Cheng Yao, et al. \"CityLifeSim: A High-Fidelity Pedestrian and Vehicle Simulation with Complex Behaviors.\" 2022 IEEE 2nd International Conference on Intelligent Reality (ICIR). IEEE, 2022.\n\nWhere the data is later attached to an engine like Carla or airsim", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Comprehensive Dynamic Modeling: OmniRe can handle various actors in urban settings, unlike most previous methods that focus mainly on vehicles.\n\nScene Graphs and Gaussian Splatting: The system uses 3D Gaussian splatting for detailed scene and object rendering, including control over each object.\n\nHuman Behavior Simulation: Through SMPL modeling, OmniRe accurately reconstructs human motions, even in cluttered environments, enabling simulations of interactions between pedestrians and vehicles.\n\nState-of-the-Art Performance: Extensive testing on datasets like Waymo and others show OmniRe significantly outperforms existing methods in terms of visual fidelity and reconstruction accuracy.", "weaknesses": "There are several limitations in OmniRe approach, which are correctly identified in the paper too.\n\nLighting Effects: OmniRe doesn’t model lighting variations explicitly. This can lead to visual inconsistencies when combining scene elements with differing lighting conditions, which may reduce realism in certain simulations. Addressing this would require additional modeling of lighting dynamics.\n\nNovel View Synthesis Limitations: OmniRe’s per-scene optimization approach struggles to generate satisfactory results when the camera view deviates significantly from the training trajectories. This could be a limitation for scenarios requiring a wide range of viewing angles, such as free navigation through the reconstructed scenes. The authors suggest incorporating data-driven priors or generative models as future work to address this.\n\nComputational Complexity: While the method achieves high-quality reconstructions, the complexity of the Gaussian Scene Graph and the joint optimization of multiple parameters (pose, appearance, etc.) require substantial computational resources. Training time per scene, though optimized for an RTX 4090 GPU, could still pose scalability issues for large datasets or continuous real-time simulation needs.\n\nChallenges with Real-Time Adaptability: The method’s reliance on SMPL modeling for human actors and per-node deformation fields, though effective, might introduce delays in real-time applications, particularly if scenes are highly dynamic or involve many non-rigid actors.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KyGrVSTKwP", "reviewer": "Reviewer_dKk5", "summary": "The paper introduces OmniRe, a novel approach for urban scene reconstruction that focuses on dynamic actors, including vehicles, pedestrians, and cyclists. OmniRe employs a Gaussian Scene Graph-based framework to model both static and dynamic objects. To address the limitations of previous methods in reconstructing non-rigid human models, OmniRe integrates SMPL for in-the-wild human representation, allowing for joint-level control. Extensive evaluations across several driving datasets demonstrate OmniRe's superior performance compared to baseline methods.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-organized and easy to follow.\n2. The proposed method for in-the-wild human representation is straightforward yet crucial for driving scene reconstruction.\n3. Both quantitative and qualitative experiments effectively support the claims made in the introduction, with OmniRe achieving state-of-the-art results across various experimental settings.", "weaknesses": "1. Handling Occlusions and Complex Dynamics: OmniRe addresses in-the-wild challenges, yet the performance might be limited by severe occlusions and overlapping actors in complex urban scenes. Further refinement or integration of advanced occlusion handling techniques could enhance reconstruction fidelity.\n2. Performance in Specific Urban Scenes: For specialized scenarios, such as highways (with fast-moving vehicles), nighttime environments, and adverse weather conditions, does OmniRe maintain high reconstruction quality under these challenging conditions?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9cwxZxJixB", "reviewer": "Reviewer_h6SA", "summary": "This paper introduces OmniRe, a comprehensive framework for dynamic urban scene reconstruction. It leverages neural scene graphs with Gaussian representations to unify the reconstruction of static backgrounds, moving vehicles, and non-rigidly dynamic actors. Additionally, it incorporates specialized designs for human modeling. The effectiveness of the approach is demonstrated across multiple datasets, showcasing superior performance in both reconstruction quality and novel view synthesis.", "questions": "1. GS-based methods generally perform well in scenarios with static environments or low vehicle speeds, as demonstrated by most of the demos on the project page. However, I am curious about the reconstruction performance of this approach in situations where the ego vehicle is moving at higher speeds.\n2. I wonder about the computational cost of reconstructing a complete segment in the Waymo dataset, as the entire pipeline seems a bit complex.\n3. Why does it seem that the reconstruction quality of NuPlan is significantly worse than that of other datasets?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written, with illustrative figures that are easy to understand. The experiments are comprehensive.\n2. Modeling dynamic objects and simulating interactive behaviors are essential for closed-loop simulation in autonomous driving systems.\n3. This work is highly engineering-oriented and demonstrates impressive results. Additionally, the authors have committed to open-sourcing the code, which will have significant value in advancing autonomous driving simulation in the future.", "weaknesses": "As mentioned by the authors in the limitations section, there are still two key shortcomings: 1. The lack of lighting modeling results in unnatural object insertions. 2. The synthesis of new viewpoints is constrained to the original trajectory, limiting the approach from achieving fully free-trajectory digital reconstruction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)