Dynamic 3D Gaussian Fields for Urban Areas
Authors: Tobias Fischer
Jonas Kulhanek
Samuel Rota Bulò
Lorenzo Porzi
Marc Pollefeys
Peter Kontschieder
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Neural Rendering
Gaussian Splatting
Dynamic Urban Areas
Primary area: machine_vision
TL;DR: Given a set of heterogeneous input sequences of a common geographic area, we optimize a single dynamic scene representation that permits rendering of arbitrary viewpoints and scene configurations at interactive speeds.

Abstract: We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200x in rendering speed.

Dynamic 3D Gaussian Fields for Urban Areas
Tobias Fischer'
Jonas Kulhanek 1.3
Samuel Rota Bulò²
Lorenzo Porzi
Mare Pollefeys'
Peter Kontschieder
ETH Zürich 2 Meta Reality Labs 3 CTU Prague
github. io/pub/Adgt/
Inputs
Geometry
Novel views in varying conditions
3D Gaussians
Dynamic Composition
Appearance
--000-
Neural Fields
Figure 1: Summary. Given set of heterogeneous input sequences that capture common geographic area in varying environmental conditions (e.g. weather, season, and lighting) with distinct dynamic objects (e.g. vehicles, pedestrians, and cyclists), we optimize a single dynamic scene representation that permits rendering of arbitrary viewpoints and scene configurations at interactive speeds.
Abstract
We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality closed-loop simulation due their limited visual quality and non-interactive rendering speeds. Recently, rasterization based approaches have achieved high-quality NVS impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as compact and flcxible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on local level via deformations. This decomposed approach enables flexible scene composition suitable for real- world applications. experiments, we surpass the state-of-the-ar by over 3 dB in PSNR and more than x00 in rendering speed.
Introduction
The problem of synthesizing novel views from a set of images has received widespread attention in recent years due to its importance for technologies like AR/VR and roboties. In particular, obtaining interactive, high-quality renderings of large-scale, dynamic urban areas under varying weather, lighting, and seasonal conditions is key requirement for closed-loop robotic simulation and immersive VR experiences. To achieve this goal, sensor-equipped vehicles act as a frequent data
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

source that is becoming widely available in city-scale mapping and autonomous driving, creating the possibility of building up-to-date digital twins of entire cities. However, modeling these scenarios is extremely challenging as heterogeneous data sources have to be processed and combined: different weather, lighting, seasons, and distinct dynamic and transient objects pose significant challenges to the reconstruction and rendering of dynamic urban areas.
In recent years, neural radiance fields have shown great promise in achieving realistic novel view synthesis of static [1 2 3 and dynamic scenes 5 77. While earlier methods were limited
to controlled environments, several recent works have explored large-scale, dynamic areas [8] g
10]. Among these, many works resort to removing dynamic regions and thus produce partial
reconstructions [0, [3] 14]. In contrast, fewer works model scene dynamics [[5] [6 17].
These methods exhibit clear limitations, such as rendering speed which can be attributed to the high cost of ray traversal in volumetric rendering.
Therefore, rasterization- based techniques [8] 19
have recently emerged as a viable alternative.
Most notably, Kerbl et al. [18] propose a scene representation based on 3D Gaussian primitives that can be efficiently rendered with a tile-based rasterizer at a high visual quality. While demonstrating impressive rendering speeds, it requires millions of Gaussian primitives with high-dimensional spherical harmonics coefficients as color representation to achieve good view synthesis results. This limits its applicability large-scale urban areas due high memory requirements. Furthermore, due to its explicit color representation, it cannot model transient geometry and appearance variations commonly encountered city-scale mapping and autonomous driving use cases such as seasonal and weather changes. Lastly, the approach is limited to static scenes which complicates representing dynamic objects such as moving vehicles or pedestrians commonly encountered urban areas.
To this end, we propose 4DGF, a method that takes a hybrid approach to modeling dynamic urban areas. In particular, we use 3D Gaussian primitives as an efficient geometry scaffold. However, we do not store appearance as a per-primitive attribute, thus avoiding more than 80% of its memory footprint. Instead, we use fixed-size neural fields compact and flexible alternative. This allows us model drastically different uppearances and transient geometry which is essential to reconstructing urban areas from heterogeneous data, Finally, we model scene dynamics with graph- based representation that maps dynamic objects to canonical space for reconstruction. We model non-rigid deformations in this canonical space with our neural fields to cope with articulated dynamic objects common in urban areas such as pedestrians and cyclists. This decomposed approach further enables a flexible scene composition suitable to downstream applications. The key contributions of this work are:
We introduce 4DGF, hybrid neural scene representation for dynamic urban areas that leverages 3D Gaussians as an efficient geometry scaffold and neural fields as a compact and flexible appearance representation.
We use neural fields to incorporate scene-specific transient geometry and appearances into the rendering process of 3D Gaussian splatting, overcoming its limitation to static, homogeneous data sources while benefitting from its efficient rendering.
We integrate scene dynamies via i) a graph-based representation, mapping dynamic objects to canonical space, and ii) modeling non-rigid deformations in this canonical space. This enables effective reconstruction of dynamic objects from in-the-wild captures.
We show that 4DGF effectively reconstructs large-scale, dynamic urban areas with over ten thousand images, achieves state-of-the-art results across four dynamic outdoor benchmarks 21 22
23
and is more than 200x faster to render than the previous
2 Related Work
Dynamic scene representations. Scene representations are pillar of computer vision and graphies research [24]. Over decades, researchers have studied various static and dynamic scene representations
for numerous problem setups [2] 27 28 30
B1
32 [3
34] Recently, neural rendering (35)
has given rise to a new class of scene representations for photo reato-te image synthesis. While earlier
methods this scope were limited to static scenes 2 3 36
37
38) dynamic scene representations
have emerged quickly 4 These scene representations can be broadly classified into implicit and
explicit representations. Implicit representations (5] 6 7 39
16] encode the scene as a
parametric function modeled as neural network, while explicit representations
42 43
45
use

---

a collection of low-level primitives. In both cases, scene dynamics are simulated as i) deformations ii) particle-level motion such as scene flow
of canonical volume [5]
42 39
16 46]
or iii) rigid transformations of local geometric primitives [4 On the contrary. traditional computer graphics literature uses scene graphs to compose entities into complex scenes [47] Therefore, another
area of research explores decomposing scenes into higher-level elements [3] 48
32
15
49
50, 17],
where entities and their spatial relations are expressed as a directed graph. This concept was recently revisited for view synthesis [1 [7]. In this work, we take hybrid approach that uses i) explicit geometric primitives for fast rendering, ii) implicit neural fields to model appearance and geometry variation, and iii) a scene graph to decompose individual dynamic and static components.
Efficient rendering and 3D Gaussian splatting. Aside from accuracy, the rendering speed of a scene representation is equally important. While rendering speed highly depends on the representation efficiency itself, it also varies with the form of rendering that is coupled with to generate an image [51] Traditionally, neural radiance fields K use implicit functions and volumetric rendering which produce accurate renderings but suffer from costly function evaluation and ray traversal. To
remedy these issues, many techniques for caching and efficient sampling [22 53 54
36
55 [7 have
been developed. However, these approaches often suffer from excessive GPU memory requirements [52] and are still limited in rendering speed [54] 55 17.. Therefore, researchers have opted to exploit more efficient forms of rendering, baking neural scene representations into meshes for efficient rasterization [19 [I] This area of research has recently been disrupted by 3D Gaussian splatting [8]. which i) represents the scene as set of anisotropic 3D Gaussian primitives ii) uses an efficient tile-based, differentiable rasterizer, and iii) enables effective optimization by adaptive density control (ADC), which facilitates primitive growth and pruning. This led to a paradigm shift from baking neural scene representations
more streamlined approach.
However, the method of Kerbl. et al. [8] exhibits clear limitations, which has sparked a very active 59
field of research with many concurrent works (6. 57 58
60,
61
62
63
64
655 For instance,
several works tackle dynamic scenes by adapting approaches described in the paragraph above 44
66
68 69 70). Another line of work focuses modeling larger-scale scenes [65
71 72 Lastly,
several concurrent works investigate the reconstruction of dynamic street scenes 73
74 75 These
methods are generally limited to homogeneous data and in scale. In contrast, our method scales to tens of thousands of images and effectively reconstructs large, dynamic urban areas from heterogeneous data while also providing orders of magnitude faster rendering than traditional approaches.
Reconstructing urban areas. Dynamic urban areas are particularly challenging to reconstruct due to the complexity of both the scenes and the capturing process. Hence, significant research efforts have focused on adapting view synthesis approaches from controlled, small-scale environments to larger, real-world scenes. In particular, researchers have investigated the use of depth priors from e.g. LIDAR, providing additional information such as camera exposure, jointly optimizing camera parameters, and developing specialized sky and light modeling approaches [8]
9
[0
12
[3] [4]
However, since scene dynamics are challenging to approach, many works simply remove dynamic areas, providing only a partial reconstruction. A few works explicitly model scene dynamies. but
suffer from limitations in terms of scalability 15 49 45
accuracy [66. rendering speed 17 76 77),
17. We introduce a mechanism to handle transient geometry and varying appearance, improve rendering efficiency, and, inspired by 45 771. propose an approach model the local articulated motion of non-rigid dynamic objects without using semantic priors. Consequently, our work enables the reconstruction of much larger urban areas with a significantly higher number and diversity of dynamic objects across multiple in-the-wild captures.
or modeling of non-rigid and uncommon objects [15
49
45
how global rigid object motion is handled in 15
49
3 Method
3.1 Problem setup
We are given set of heterogeneous sequences S that capture common geographic area from moving vehicle. The vehicle is equipped with calibrated cameras mounted in surround-view setup. We denote with C, the set of cameras of sequence E S and with C' the total set of cameras, i.e. C' = U>es C.. For each camera e E C, we assume to know the intrinsic K. parameters and the pose P. E SE(3), expressed in the ego-vehicle reference frame. Ego-vehicle poses P', E SE(3) are provided for each sequence e S and timesteps t E T. and are expressed in the world reference
3

---

(V, Representation (Sec 3.2)
Composition & Rendering (Sec 3.3)
Geometry scaffold
Neural Fields
Static
positions μ opacities covariances
corr.
Dynamic colors deform, 8
V
3D Gaussians
Static Scene
Dynamic Objects
configuration
v
8 [R||]
Dynamic Composition Rendered View
Figure 2: Overview. To render an image of sequence a at time f, we first evaluate the scene graph G = (V,E) which stores latent codes w its nodes V and coordinate transformations [R|t] at its edges &, i.e. the configuration of the dynamic objects and the overall scene. We then the scene configuration to determine the active sets 3D Gaussians The 3D Gaussians G' and latent codes serve as conditioning signals to the neural fields & and ıb. which output, for each 3D Gaussian ga E G. an appearance conditioned color c an opacity correction term V/2 for static Gaussians modeling transient geometry. and a dynamic deformation 8% for non-rigt non-rigid dynamic 3D Gaussians modeling pedestrians. Finally, the retrieved information is used to compose set of 3D Gaussians that represent the dynamic scene at (s, from which render the image.
frame that is shared across all sequences. Here, T, denotes set of timestamps relative 8. Indeed, we assume that timestamps cannot be compared across sequences because we lack a mapping to a global timeline, which is often the case with benchmark datasets due to privacy reasons. For each sequence E S, camera c E C. and timestamp E Т, we have an RGB image I( E [0, 1|2 [
Each sequence has additionally an associated set of dynamic objects Oz. Dynamic objects E O, are associated with a 3D bounding box track that holds (stationary) 3D object dimensions S, E F. and poses {Eto .... < SE(3) w.r.t. the ego-vehicle frame, where t, T. T. Our goal is to estimate the plenoptic function for the shared geographic area spanned by the training sequences, i.e. a function (P, K, t, s), which outputs a rendered RGB image of size (H, W) for given camera pose P with calibration K in the conditions of sequence E S at time t E T..
3.2 Representation
We model parameterized, plenoptic function fo, which depends the following components: a scene graph G that provides the scene configuration and latent conditioning signals w for each sequence object and time f, ii) sets of 3D Gaussians that serve geometry scaffold for the scene and objects, and iii) implicit neural fields that model appearance and modulate the geometry scaffold according the conditioning signals. See Figure for an overview of method.
Scene configuration. Inspired by [17, we factorize the scene with a graph representation holding latent conditioning signals at the nodes V and coordinate system transformations along the edges E. The nodes V consist a root node V, defining the global coordinate system, camera S. sequence nodes {uī hez, and dynamic object nodes
nodes {Veleec. and for each sequence {v. We associate latent vectors w to sequence and object nodes representing local appearance and geometry. Specifically, model the time-varying sequence appearance and geometry via
(t)) Gt)(t))
where Ag and G, are appearance and geometry modulation matrices, respectively, and 7-) is a ID basis function of sines and cosines with linearly increasing frequencies at log-scale [35]. Time t is
normalized to [-1, 1] via the maximum sequence length max,es [Ta]- For objects, we use both an object code and time encoding
3 =
(2)
Nodes in the graph G connected by oriented edges that define rigid transformations between the canonical frames of the nodes. We have P for sequence to root edges, P. for camera to sequence edges, and & for object to sequence edges.
3D Gaussians. We represent the scene geometry with sets of anisotropic 3D Gaussians primitives G L oe Oz, 8}. Each 3D Gaussian primitive gı is parameterized by its mean

---

Mg. covariance matrix Σg, and a base opacity The covariance matrix is decomposed into rotation matrix represented as unit quaternion 94 and scaling vector ag E IR.. The geometry of gu. is represented by
g (x) cxp (-2 u4l / [4a)
(3)
The common scene geometry scaffold is modeled with a single set of 3D Gaussians G,, while we have a separate set Go of 3D Gaussians for each dynamic object o. Indeed, scene geometry is largely consistent across sequences while object geometries are distinct. The 3D Gaussians G, are represented in world frame, while each set G. is represented in a canonical, object-centric coordinate frame, which can be mapped to the world frame by traversing G.
Differently from [8] our 3D Gaussians do not hold any appearance information, reducing the memory footprint of the representation by more than 80%. Instead, we leverage neural fields to regress color information cf and updated opacity of for each sequence S and time E T. For 3D Gaussians in G, modeling the scene scaffolding, we predict an opacity attenuation term of that is used to model transient geometry by downscaling org. Instead, for 3D Gaussians in G. modeling objects the base opacity is left invariant. Hence
f1x aid a
if E G,
0%
5.4 =
(4)
ak else.
The attenuation term enforces a high base opacity for every 3D Gaussian visible in al least one sequence. Therefore, we can obtain pruning decisions in ADC by thresholding the base opacity ag- which is directly accessible without computational overhead, without risking the removal of transient geometry.
Furthermore, in the presence of non-rigid objects o, we predict deformation terms & E TR3 to the position of 3D primitives in G. via the neural fields, for each time E Tg. In this case, the position of the primitive in object-centric space at time / is given by
14 + +%
(5)
Appearance and transient geometry. Given the scene graph G and the 3D Gaussians G, we use two neural fields to decode the aforementioned parameters for each primitive. In particular, for 3D Gaussians in G, modeling the static scene, the neural field is denoted by & and regresses the opacity attenuation term vi and color a.t given the 3D Gaussian primitive's position 14g, a viewing direction d, the base opacity aş and the latent code of the node w/, i.e.
(M c = ( d, akk w!)
where E S and / T., Note that since the opacity attenuation ve contributes to modeling transient geometry by removing parts the scene encoded the original set of Gaussians, it does not depend on the viewing direction d.
For 3D Gaussians in G, modeling dynamic objects, the neural field is denoted by şi and regresses color c Besides the primitive's position and viewing direction, we condition e on latent vectors wi and w/ to model both local object texture and global sequence appearance such as illumination. Here, the sequence is the one where o belongs to, i.e. satisfying E O., and E T,, Accordingly, the color cx for 3D Gaussian G2 given by
sut == (14,, d. W w() c
Both i4 and d are expressed in the canonical, object-centric space of object o. Using neural fields has three key advantages for our purpose. First, by sharing the parameters of d and across all 3D Gaussians G, we achieve a significantly more compact representation than in [18] when scaling to large-scale urban areas. Second, it allows to model sequence sequence-dependdnt appearance and transient geometry which is fundamental to learning a scene representation from heterogeneous data. Third, information sharing between nodes enables interaction of sequence and object appearance.
However, querying a neural field is more complex than a spherical harmonics function as Therefore, we i) use efficient hash-grid representations [53] to minimize query complexity and, ii)
5

---

carefully optimize the rendering workflow to minimize the amount of queries. In particular, we skip out-of-view 3D Gaussians and implement vectorized query function to that retrieves the parameters of all relevant dynamic objects in parallel. We refer to Section 4.3 for runtime analysis.
Non-rigid objects. Street scenes are occupied not only by rigidly moving vehicles but also by, e,g,, pedestrians and cyclists that move in a non-rigid manner. These pose significant challenge due to their unconstrained motion under limited visual coverage. Therefore, we take a decomposed approach to modeling non-rigid objects. First, we represent the local articulated motion of non-rigid objects like pedestrians as deformation in canonical space. We use deformation head X that predicts a local position offset 0% via
(8) given an intermediate feature representation f. of sb conditioned on f4 and time t. We deform the position of μ4: over time in canonical space as per Equation (5) Second, we use the scene graph G to model the global rigid object motion, transforming the objects from object-centric world space with rigid body transformation. We use a general design to cover a wide range of scenarios, such as pedestrians holding a stroller or shopping bags, cyclists, and animals. See Figure I0 in our supp. mat.
81 = x(fs,
Background modeling. To achieve a faithful rendering of far-away objects and the sky, it is important to have a background model. Inspired by (54). where points are sampled along ray at increasing distance outside the scene bounds, we place 3D Gaussians on spheres around the scene with radius r for i 21,2.3) where is half of the scene bound diameter. To avoid ambiguity with foreground scene geometry and to increase efficiency, we remove all points that are i) below the ground plane, ii) occluded by foreground scene points, iii) outside of the view frustum of any training view. To uniformly distribute points on each sphere, we utilize the Fibonacci sphere sampling algorithm (78), which arranges points in spiral pattern using golden patio-based formula. Even though this sampling is not optimal, it serves as a faster approximation of the optimal sampling.
3.3 Composition and Rendering
Scene composition. To render our representation from the perspective of camera c at time / in sequence 8, we traverse the graph to obtain the latent vector w and the latent vector we of each visible object E Og, i.e. such that / E Tg. Moreover, for each 3D Gaussian primitive 9& in G, we use the collected camera parameters, object scale, and pose information to determine the transformation II mapping points from the primitive's 's reference frame (e.g. world for Gr, object-space for Go) to the image space of camera c. Opacities of are computed per Equation (4), while colors ci. are computed for primitives in G, and in Go via Equations (6) and (7). respectively. For non-rigid objects in Go, we compute the primitive positions μ via Equation
Rasterization. To render the scene from camera c, we follow [18] and splat the 3D Gaussians to the image plane. Practically, for each primitive, we compute 2D Gaussian kernel denoted by gf, with mean 14g given by the projection of the primitive's position to the image plane, i.e. 1a == IT (MA). and with covariance given by Σ == J& where Jg. is the Jacobian of IIğ evaluated at p4g Finally, we apply traditional alpha compositing of the 3D Gaussians to render pixels p of camera c:
K
k-1
(p)
= Σ
wk
(1
with
Wk = ag (p)
1=
3.4 Optimization
To optimize parameters of fo. i.e. 3D Gaussian parameters μ4: 0gg, Qk and ag. sequence latent vectors wi and implicit neural fields 1a and o, we use an end-to-end differentiable rendering pipeline. We render both RGB color image i and depth image D and apply the following loss function:
I(i,,, Angh I) + Assim Essim (i, I) + Adep (D,D) D)
(10)
where Ligb is the L1 norm, Lssim is the structural similarity index measure [79], and Loep is the L2 norm. We use the posed training images and LiDAR measurements as the ground truth. If no depth ground-truth is available,
drop the depth-related loss from L.
Pose optimization. Next to optimizing scene geometry, is crucial to refine the pose parameters of the reconstruction for in-the-will wild scenarios since provided poses often have limited accuracy [10 [7].

---

Method
Residential
Downlown
Render (6)
PSNR
SSTM
LPIPS /
PSNR 1
SSIM
LPIPS
PSNR
SSIM
I.IPS
Nerfacto Emb.
19.83
0.637
0.562
18.05
0.655
0.625
18.94
0.646
0.594
11.5
Nerfacto Emb. Time
20.05
0.641
0.562
18.66
0.656
0.603
19.36
0.654
0,583
11.6
SUIDS
21.76
0.556
19.91
0.601
74.0
NSG 177
22.29
0.678
0.523
20.01
0.681
0.586
21.15
0.680
0.555
21.7
4DGF (Ours)
25.78
24.16
24.97
Table 1: Novel view synthesis Argoverse [81]. Our method improves substantially over the state-of the-art while heing more than 2 faster render at the original 1550 2048 resolution.
Thus, we optimize the residuals AP', E se(3), AP. E se(3) and AC se(2) jointly with parameters 0. We constrain object pose residuals to se( to incorporate the prior that objects move on the ground plane and are oriented upright. See our supp. mat. for details on camera pose gradients.
Adaptive density control. To facilitate the growth and pruning of 3D Gaussian primitives, the optimization of the parameters 0 is interleaved by an ADC mechanism [18]. This mechanism is essential to achieve photo-realistic rendering. However, it was not designed for training on tens of thousands of images, and thus we develop streamlined multi-GPU version of it. We accumulate statistics across processes and, instead of running ADC on GPU and synchronizing the results, synchronize only non-deterministic parts of ADC, i.e. the random samples drawn from the 3D) Gaussians that are being split. These are usually much fewer than the total number of 3D Gaussians and thus avoids communication overhead. Next, the 3D Gaussian parameters are replaced by their updated replicas. However, this will impair the synchronization of the gradients because, in Py PyTorch DDP 80), parameters are only registered once at model initialization. Therefore, we re-initialize the Reducer upon finishing the ADC mechanism in the low-level API provided in [80].
Furthermore, urban street scenes pose some unique challenges ADC, such large variation in scale, e.g. extreme close-ups of nearby cars mixed with far-away buildings and sky. This can lead to blurry renderings for close-ups due to insufficient densification. We address this by using maximum 2D screen size as a splitting criterion! ! In addition, ADC considers the world-space scale aş of 3D Gaussian to prune large primitives which hurts background regions far from the camera. Hence, we first test if a 3D Gaussian is inside the scene bounds before pruning it according to ag- Finally, the scale of urban areas leads memory issues when the growth of 3D Gaussian primitives is unconstrained. Therefore, we introduce threshold that limits primitive growth while keeping pruning place. See our supp. mat. for more details and analysis.
Experiments
Datasets and metrics. We evaluate our approach across various dynamic outdoor benchmarks. First, we utilize the recently proposed NVS benchmark [17 of Argoverse 2 [8] to compare against the state-of-the-an in the multi-sequence scenario and to showcase the scalability of method. Second, we use the established Waymo Open (33. KITTI (2 and VKITTI2 (22) benchmarks to compare to existing approaches in single-sequence scenarios. For Waymo, we use the dynamic-32 split of (7)), while for KITTI and VKITTI2 we follow [16] 17 We apply commonly used metrics to measure view synthesis quality: PSNR, SSIM 77). and LPIPS (AlexNet) 22
Implementation details. We use Angb == 0.8, Assim == 0.2 and Adept == 0.05. We use the LiDAR point clouds as initialization for the 3D Gaussians. We first filter the points of dynamic objects using the 3D bounding box annotations and subsequently initialize the static seene with the remaining points while using the filtered points initialize each dynamic object. We use mean voxelization with voxel size 7 to remove redundant points. See our supp. mat. for more details.
4.1 Comparison to state-of-the-art
We compare with prior art across two experimental settings: single-sequence and suutieseeuuece. In the former, we are given a single input sequence and aim to synthesize hold-out viewpoints from that same sequence. In the latter, we are given multiple, heterogeneous input sequences and aim to synthesize hold-out viewpoints across all of these sequences from a single model.
Note that while this criterion was described in [8] it used in the experiments.
7

---

KITTI [75%]
KITTI [50%]
KITTI [25%)
Method
PSNR 1
SSIM 1
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
NSG
15
21.53
0.673
0.254
21.26
0.659
0.266
20.00
0.632
0.281
SUDS 16
22.77
0.797
0.171
23.12
0.821
0.135
20.76
0.747
0.198
MARS
83
24.23
0.845
0.160
24.00
0.801
0.164
23.23
0.756
0.177
StreetGaussians
73
25.79
0.844
0.081
25.52
0.841
0.084
24.53
0.824
0.090
MI MINN
17
28. 58
0.907
0.052
27.51
0.898
0.055
26.51
0.887
0.060
4DGF (Ours)
31.34
0.945
0.026
30.55
0.931
0.028
29.08
0.908
0.036
VKITT12 [75%]
VKITT12 [50%]
VKITTI2 [25%]
Method
PSNR 1
SSIM 1
LPIPS
PSNR
SSIM
LPIPS
PSNK
SSIM
LPIPS J
NSG
15
23.41
0.689
0.317
23.23
0.679
0.325
21.29
0.666
0.317
SUDS
16
23.87
0.846
0.150
0.142
22.18
MARS 83
29.79
0.917
0.088
29.63
0.916
0.087
27.01
0.887
0.104
StreenGasssians 7
30.
0.935
0.025
29.91
0.932
0.026
28.52
0.917
0.034
ML-NSG
[+
29.73
0.912
0.065
29.19
0.906
0.066
28.29
0.901
0.067
4DGF (Ours)
30.67
0.943
0.035
30.45
0.939
0.036
29.27
0.923
0.041
Table 2:
Novel view synthesis KITTI [21) and VKITTI2
22].
Our method produces state-of-the-art results
across both henchmarks and at varying training view fractions.
Multi-sequence setting. In Table we show results on the Argoverse 2 NVS benchmark proposed in [17]. We compare to state-of-the-art approaches [16] [7] and the baselines introduced in [7]. The results highlight that our approach scales well to large-scale dynamic urban scenes, outperforming previous work in performance and rendering speed by significant margin. Specifically, we outper- form [17 by more than 3 dB in PSNR while rendering more than 200 faster. To examine these results more closely, we show qualitative comparison in Figure 3 We see that while SUDS [I6] struggles with dynamic objects and ML-NSG [17] produces blurry renderings, our work provides sharp renderings and accurately represented dynamic objects. in both RGB color and depth images. Overall, the results highlight that our model can faithfully represent heterogeneous data at high visual quality in single 3D representation while being much faster to render than previous work.
Single-sequence setting. In Table 2. we show results on the KITTI (21] and VKITTI [22) benchmarks at vary- ing training view fractions, iee. from dense towards sparse follow the experimental protocol in [77) and show an additional comparison on the KITTI dataset with different data split that uses also ap- prox. 50% of views for training in Table
Method
PSNR
SSIM
LPIPS
SUDST MARS 83
16
83
23.12
0.821
0.135
24.00
0.801
0.164
view setting. Furthermore,
NeuRAD 77
27.00
0.795
0.082
NeuRAD 2x. 77
27.91
0.822
0.066
4DGF (Ours)
30.01
0.913
0,052
3 Our approach consistently outperforms previous work as well as con- current 3D Gaussian- show results on Waymo Open (23] specifically
Table 3: Novel view synthesis on
KITTI (21) on split from (77). We follow
approaches (73.. In Table the
the experimental protocol in [77 to compare
additional works. baseline from [
Dynamic-32 32 split proposed in (76). We outperform previous work by a large margin while our rendering speed 700x faster than the best alternative. Note that our rendering speed increases for smaller-scale scenes. Furthermore, we show that, contrary to previous approaches, our method does not suffer from lower view quality in dynamic areas. This corroborates the strength of our contributions, showing that our method is not only scalable to large-scale, heterogeneous street data but also demonstrates superior performance in smaller-scale, homogeneous street data.
4.2 Ablation studies
We verify our design choices in both multi- and single-sequence settings. For a fair comparison, we set the global max- imum of 3D Gaussians lion, respectively. We perform these ablation studies on the residential split of [17]. We use the full overlap in the dellti-sqqueeee
Full Image
Dynamic-Onil
Method
PSNR 1
SSIM
PSNR
SSIM
Render (x)
DºNeRF 41
24.17
0.642
21.44
0.494
and 4.1 mil-
HyperNeRR 39
24.71
0.682
22.43
0.554
Eeee
27.62
0.792
24.18
0.670
18.9
4DGF (Ours)
29.04
0.881
29.34
0.884
0.025
Table 4: Novel view synthesis on Waymo Open [2]]. We use the Dynamic-32 split (76). Contrary prior work, we do not exhibit lower view quality in dynamic areas.
multi-sequence setting, while using a sin- gle sequence of this split for the single- seguue sequence setting. In Ta, we verify the components that are not specific to the multi-sequence setting. In particular, we show that our approach to modeling scene dynamics is highly effective, evident from the large disparity in performance between the static and the dynamic variants. Next, we show that modeling appearance with neural field is par with the proposed solution in [18), while
8

---

Dynamic Neural
Background
SSIM
LPIPS 4
PSNR
SSIM
I.PIPS
24.31
0.814
0.287
8B
22.37
0.741
28.55
0.839
0.262
8.6 GB
25.13
0.767
0.412
28. 49
0.838
0.262
4.5
25.78
0.772
0.405
(a) Single-sequence.
(b) sequinee
Table 5: Ablation studics. We show that (a) approaches modeling scene dynamics and background regions are effective and neural fields are on-par with spherical harmonics while more memory efficient to train, and (b) using implicit fields for appearance and geometry is crucial for the multi seulti-s setting. We control for the maximum number of 3D Gaussians for fair comparison.
Dataset
Arguverse [8] 1550 2048
Wayme Open 23
Resolution
640 960
Mean (ms)
Percentage (%)
Avg. number of 3D Gaussians
8.02M
2.75M
1. Scene graph evaluntion: retrieve u. [Rt]]]. 3D Gaussian at
38.5
13.0
25.75
52.4
Scene composition: apply [R]t]] 3D Gaussians
1.90
3. 3D Gaussian projection
2.0
3,5
2.75
5.6
Query ncural fields
6.55
5. Rasterization
21.3
3.0
12.15
24.8
73.6
FPS
13.6
20.4
Table 6: Inference runtime analysis. We report the individual and average timings of our method's components datasets. Overall, scene graph evaluation (1.) and rasterization (S.), dominate the runtime. While total runtime correlates with scene scale and image resolution, we achieve interactive frame rates on both datasels.
being more memory efficient. In particular, when modeling view-dependent color as a per-peurue Gaussian attribute as in [88] the model uses 8.6 GB of peak GPU memory during training, while it uses only 4.5 GB with fixed-size neural fields. Similarly, storing the parameters of the former takes 922 MB, while the latter takes only 203 MB. Note that this disparity increases with the number of 3D Gaussians per scene. Finally, we achieve the best performance when adding the generated 3D Gaussian background.
We now scrutinize components specific to multi-sequence data in Table 5b. We compare the view synthesis performance of our model when i) not modeling sequence appearance or transient geometry, ii) only modeling sequence appearance, iii) modeling both sequence appearance and transient geometry. Naturally, we observe large gap in performance between i) and ii), since the appearance changes between sequences are drastic (see Figure 3) However, there is still a significant gap between ii) and iii), demonstrating that modeling both sequence appearance and transient geometry is important for view synthesis from heterogeneous data sources. Finally, we provide qualitative results non-rigid object view synthesis in Figure 4 and show that our approach can model articulate motion without the use of domain priors. In supp. mat., provide further analysis.
4.3 Runtime analysis
We divide our algorithm into its main components and report the individual inference runtimes across two datasets in Table 6 While the runtime clearly correlates with scene complexity and image resolution, we observe that, on average, the runtime is dominated by scene graph evaluation and rasterization, accounting for more than 75% of the total runtime. This owes to the complexity of rasterizing millions of primitives across a high-resolution image which is computationally demanding even for efficient rasterization algorithms [8). and handling hundreds objects across one or multiple dynamic captures, making the retrieval of the 3D Gaussians and latent codes costly. In contrast, the queries to the neural fields account for only 13.3% of the average total runtime, making it a viable alternative to the spherical harmonics function in [18]. Overall, our method achieves interactive rendering speeds on both datasets and 20.4 FPS on average.
thousands of dynamic
5 Conclusion
presented 4DGF, a neural scene representation for dynamic urban areas. 4DGF models highly dynamic, large-scale urban areas with 3D Gaussians as efficient geometry scaffold and compact but flexible neural fields modeling large appearance and geometry variations across captures. We use scene graph to model dynamic object motion and flexibly compose the representation at arbitrary

---

Figure 3: Qualitative results on Argoverse 81. Our method produces significantly sharper renderings both in foreground dynamic and static background regions, with much fewer artifacts egg. in arcas with transient geometry such tree branches (left). Best viewed digitally.
Figure 4: Qualitative results on Waymo Open our model (top-left to bottom-right). As the woman (marked with a red box) gets out of the car and walks away, we successfully model her articulated motion and changing body poses.
23).
show a sequence evaluation views synthesized by
configurations and conditions. We jointly optimize the 3D Gaussians, the neural fields, and the scene view synthesis quality and interactive rendering speeds.
graph, showing
Limitations and future work. While 4DGF improves novel view synthesis in dynamic urban areas, the challenging nature of the problem leaves room for further exploration. Although we model scene dynamics, appearance, and geometry variations, other factors influence image renderings in real-world captures. First, in-the-wild captures often exhibit distortions caused by the physical image formation process. Therefore, modeling phenomena like rolling shutter, white balance, motion and defocus blur, or chromatic aberrations is necessary to avoid reconstruction artifacts. Second, the assumption of pinhole camera model in [88] persists in our work. Thus, our method falls short of modeling more complex camera models like equirectangular cameras and other sensors such as LiDAR, which may be limiting for certain capturing or simulation settings.
Broader impact. We expect our work to positively impact real-world use cases like robotic simulation and mixed reality by improving the underlying technology. While we do not expect malicious uses of our method, note that an inaccurate simulation, i.e. failure of our system, could misrepresent the robotic system performance, possibly affecting real-world deployment.
10

---

6 Acknowledgements
The authors thank Haithem Turki, Songyou Peng, Erik Sandström, François Darmon, and Jonathon Luiten for useful discussions. Tobias Fischer was supported by a Meta SRA.
References
[1] L.. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, "Oecupancy networks: Learning 3d reconstruction in function space," in CVPR, 2019.
[2] V. Sitzmann, M. Zollhöfer, and Wetzstein, "Scenc representation networks: Continuous 3d-structure- aware neural scene representations. NeurlPS, 2019.
[3] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, Ramamoorthi, and R. Ng. "Nerf: Representing scenes as neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
7. Li, Niklaus, N. Snavely, and O. Wang, "Neural scene flow fields for space-time view synthesis of dynamic scenes," in CVPR, 2021.
[5] A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer, "D-nerf: Neural radiance fields for dynamic scenes," in CVPR, 2021.
[6] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt, "Non-rigid neural radiance fields: Reconstruction and novel view synthesis of dynamic scene from monocular video," in ICCV, 2021.
[7] W. Xian, J.-B. Huang, J. Kopf, and C. Kim, "Space-time neural irradiance fields for free-viewpoint video,"
in CVPR, 2021.
[8] R. Martin-Brualla, N. Radwan, S. Sajjadi, J. T. Barron, A. Dosovitskiy, and Duckworth, "Nerf in the wild: Neural radiance fields for unconstrained photo collections," in CVPR, 2021.
[9] K. Rematas, A. Liu, P. Srinivasan, J. Barron, Tagliasacchi, Funkhouser, and V. Ferrari, "Urban radiance fields," CVPR, 2022.
[10] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. Srinivasan, T. Barron, and H. Kretzschmar, "Block-nerf: Scalable large scene neural view synthesis," " in CVPR, 2022.
[11] J. Y. Liu, Y. Chen, Z. Yang. J. Wang, S. Manivasagam, and R. Urtasun, "Real-time neural rasterization for large scenes," in ICCV, 2023.
[12] 7. Xie, Zhang, W. Li, Zhang, and Zhang, "S-nerf: Neural radiance fields for street views," arXiv preprint arXii:200330074 2023.
[13] Z. Wang. T. Shen, Gao, Huang, J. Munkberg, J. Hasselgren, Gojeic, W. Chen, and Fidler, "Neural fields meet explicit geometric representations for inverse rendering of urban scenes," " CVPR, 2023.
[14] V. Rudnev, M. Elgharib, W. Smith, L. Liu, V. Golyanik, and C. Theobalt, "Nerf for outdoor scene relighting," 2022.
in 2022.
[15] J. Ost, F. Mannan, N. Thuerey, J. Knodt, and F. Heide, "Neural scene graphs for dynamic scenes," in CVPR, 2021.
[16] H. Turki, Y. Zhang, F. Ferroni, and D. Ramanan, "Suds: Scalable urban dynamic seenes," CVPR, 2023.
[17] T. Fischer, L. Porzi, Rota Buld, M. Pollefeys, and Kontschieder, "Multi- level neural scene graphs for dynamic urhan environments," in CVPR, 2024.
[18] B. Kerhl, G. Kopanas, T. Leimkühler, and G. Drettakis, "3d gaussian splatting for real-time radiance field rendering," " ACM Transactions on Graphics, vol. 42, no. 2023.
[19] Z. Chen, T. Funkhouser, P. Hedman, and A. Tagliasacchi, "Mobilenerf: Exploiting the polygon rasterization pipeline for efficient neural field rendering mobile architectures," in CVPR, 2023.
[20] F. Lu, Y. Xu, G. Chen, H. Li, K.-Y. Lin, and C. Jiang. "Urban radiance field representation with deformable neural mesh primitives," in ICCV, 2023.
[21] A. Geiger, P. Lenz, and Urtasun, "Are ready for autonomous driving? the kitti vision benchmark suite," in CVPR, 2012.
[22] Y. Cabon, N. Murray, and M. Humenberger, "Virtual kitti 2," 2020.
[23] P. Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Y. Chai, B. Caine, et al., "Scalability in perception for autonomous driving: Waymo open dataset." in Proceedings of the 2020.
IEEE/CVF conference computer vision and pattern recognition, pp.
11

---

[24] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard, "Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age." IEEE Transactions on robotics, vol. 32, no. 6. pp. 1309-1332, 2016.
[25] C. Shen, J. F. O'Brien, and J. R. Shewchuk. "Interpolating and approximating implicit surfaces from polygon soup." in ACM SIGGRAPH 2004 Papers, SIGGRAPH '04, p. 896 904, 2004.
[26]
Bloomenthal and B. Wyvill, "Introduction to implicit surfaces," 1997.
[27] M. Kaess, "Simultaneous localization and mapping with infinite planes." 2015 IEEE International Confer- ence on Robotics and Automation (ICRA). pp- 4605-4611, 2015.
[28] N. Fairtield, G. A. Kantor, and D. S. Wettergreen, "Real-time slam with octree evidence grids for exploration in underwater tunnels," Journal of Field Robotics, vol. 24, 2007.
[29] Y. Lu and Song, "Visual navigation using heterogencous landmarks and unsupervised geometric constraints," IEEE Transactions on Robotics, vol. 31, pp. 736-749, 2015.
[30] M. Pollefeys, D. Nistér, J.-M. Frahm, A. Akbarzadeh, P. Mordohai, B. Clipp. C. Engels. D. Gallup. S. J. Kim, P. C. Merrell, C. Salmi, S. N. Sinha, B. Talton, L.. Wang. Q Yang. H. Stewénius, R. Yang. G. Welch, and H. Towles, "Detailed real-time urban 3d reconstruction from video," " IJCV, vol. 78, pp. 143-167, 2007.
[31] R. F. Salas- Maras-Moreno, R. A. Newcombe, H. Strasdat, P. H. Kelly, and A. Davison, "Slam++: Simultaneous localisation and mapping the level of objects," in CVPR, 2013.
[32] I. Armeni, Z.-Y. Y. Hc, J. Gwak, A. R. Zamir, M. Fischer, J. Malik, and S. Savarese, "3d seene graph: A structure for unified semantics, 3d space, and camera," in ICCV, 2019.
[33] J. I Luiten, T. Fischer, and B. Leibe, "Track to reconstruct and reconstruct to track," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1803-1810, 2020.
[34] A. Schmied, T. Fischer, M. Danelljan, M. Pollefeys, and F. Yu, "R3d3; Dense 3d reconstruction of dynamic scenes from multiple cameras," in ICCV, 2023.
[35] M. Niemeyer, L. Mescheder, M. Oechsle, and A. Geiger, "Differentiable volumetric rendering: Learning implicit 3d representations without 3d supervision." in CVPR, 2020.
[36] S. Fridovich-Keil, A. Yu, M. Tancik, Q Chen, B. Recht, and A. Kanazawa, "Plenoxels: Radiance fields without neural networks," " CVPR, 2022.
[37] Q. Xu, Z. Xu, J. Philip, S. Bi, Z. Shu, K. Sunkavalli, and U. Neumann, "Point-nerf: Point-based neural radiance fields," in CVPR, 2022.
[38] G. Riegler and V. Koltun, "Stable view synthesis," in CVPR, 2021.
[39] K. Park. U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla and M. Seitz, "Hypemerr: nigher-dimeesional representation for topologically varying neural radiance fields," arXiv preprint ar arx::22066. 1e..21000.322, 13228, 2021.
[40] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang, "Dynamic view synthesis from dynamic monocular video," in CVPR, 2021.
[41] T. Wu, F. Zhong, A. Tagliasacchi. F. Cole, and C. Oztireli. "D" 2nerf: Self-supervised decoupling of dynamic and static objects from monocular video." NeurIPS, 2022.
[42] Fang, Yi, X. Wang, L. Xie, X. Zhang, Liu, M. Nießner, and Q. Tian, "Fast dynamic radiance fields with time-ave aware neural voxels," in SIGGRAPH Asia 2022 Conference Papers, 2022.
[43] B. Park and C. Kim, "Point-dyarf: Point-based dynamic radiance fields from monocular video," 2023. [44] Luiten, G. Kopanas, B. Leibe, and D. Ramanan, "Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis," in 3DV, 2024.
[45] Z. Yang, Y. Chen, Wang, S. Manivasagam, W.-C. Ma, A. J. Yang, and Urtasun, "Unisim: A neural closed-loop sensor simulator," " in CVPR, 2023.
[46] Z. Li, Q. Wang, F. Cole, R. Tucker, and N. Snavely, "Dynibar: Neural dynamic image-baser rendering," in CVPR, 2023.
[47] S. Cunningham and M. J. Bailcy, "Lessons from scene graphs: using seene graphs to teach hicrarchical modeling," Computers & Graphics, vol. 25, no. 4, pp. 703-711, 2001.
[48] S. Tulsiani, S. Gupta, D. Fouhcy, A. Efros, and Malik, "Factoring shape, pose, and layout from the
2d image a 3d scene," in CVPR, 2018.
[49] A. Kundu, K. Genova, X. Yin, A. Fathi, C. Pantofaru, L. J. Guibas, A. Tagliasacchi, F. Dellaert, and
T. Funkhouser, "Panoptic neural fields: semantic object-aware neural scene representation." in CVPR, 2022.
[50] A. Rosinol, A. Gupta, M. Abate, J. Shi, and L. Carlone, "3d dynamic scene graphs: Actionable spatial perception with places, objects, and humans." arXiv preprint arXiv:2002.0628)) 2020.
12

---

[51] A. Tewari, J. Thies, B. Mildenhall, P. Srinivasan, E. Tretschk, W. Yifan, C. Lassner, V. Sitzmann, R. Martin- Brualla, S. Lombardi, al., "Advances in neural rendering." " in Computer Graphicy Forum, vol. 41, pp. 703-735, Wiley Online Library. 2022.
[52] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin, "Fastnerf: High-fideling neural rendering at 200fips." in Proceedings of the IEEE/CVF International Conference on Computer Vision,
pp. 14346-14355, 2021.
[53] T. Müller, A. Evans, C. Schied, and A. Keller, "Instant neural graphies primitives with multiresolution hash encoding." ACM Transactions on Graphics (ToG), vol. 41, no. 4. pp. 1-15, 2022.
[54] J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman, "Mip-nerf 360: Unbounded anti-aliased neural radiance fields," in CVPR, 2022.
[55] H. Turki, D. Ramanan, and M. Satyanarayanan, "Mega-nerf: Scalable construction of large-scale nerfs for virtual thy-throughs," "" in CVPR, 2022.
[56] J. C. Lee, D. Rho, X. Sun, J. H. Ko, and E. Park, "Compact 3d gaussian representation for radiance field," arXiv preprint arXiv:2311. 2023.
[57] A. Guédon and V. Lepetit. "Sugar: Surface-aligned gaussian splatting for efficient 3d mesh reconstruction and high-quality mesh rendering." arXiv preprint arXiv:2311 122772211.777,... 2023.
[58] L. Radl, M. Steiner, M. Parger, A. Weinrauch, B. Kerbl, and M. Steinberger, "Stopphhpopp: Sorted gaussian splatting for view-consistent real-time rendering," arXiv preprint arXiv:2402.00525 2024.
[59] Z. Yu, A. Chen, B. Huang, T. Sattler, and A. Geiger, "Mip-splatting Alias-free 3d gaussian splatting," " arXiv preprint arXiv:2311.1644
2023.
[60] Z. Yu, T. Sattler, and A. Geiger, "Gaussian opacity fields: Efficient and compact surface reconstruction in unbounded scenes," arXiv preprint arXiv:2404.100 10772, 2024.
[61] O. Seiskari, J. Ylilammi, Kaatrasalo, P. Rantalankila, M. Turkulainen, J. Kannala, F. Rahtu, and A. Solin, "Gaussian splatting on the move: Blur and rolling shutter compensation for natural camera motion," arXiv preprint 2403 /3327, 2024.
[62] F. Darmon, L. Porzi, S. Rota-Buld, and P. Kontschieder, "Robust gaussian splatting," arXiv preprint arXiv:2404.04211 2024.
[63] S. R. Buld, L. Porzi, and P. Kontschieder, "Revising densification in gaussian splatting." arXiv preprint arXiv:2404.06100 2024.
[64] Y. Jiang, J. Tu, Liu, X. Gao, X. Long, W. Wang, and Y. Ma, "Gaussianshader: 3d gaussian splatting with shading functions for reflective surfaces," arXiv preprint axXiv:3311.1997,, 2023.
[65] Lin, Z. Li, X. Tang. Liu. S. Liu, Liu, Y. Lu, X. Wu, S. Xu, Y. Yan, et al., "Vastgaussian: Vast 3d gaussians for large scene reconstruction," arXiv preprint arXiv:2402.17427, 2024.
[66] Z. Yang, Gao, W. Zhou, S. Jiao, Y. Zhang, and Jin, "Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction," arXiv preprint arXiv:2309. 13101, 2023.
[67] G. Wu, T. Yi. J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang, "4d gaussian splatting for real-time dynamic scene rendering," " arXiv preprint arXiv:23/0.08528, 2023.
[68] D. Das, C. Wewer, R. Yunus, F. Ilg, and J. F. Lenssen, "Neural parametric gaussians for monocular non-rigid object reconstruction." arXiv preprint arXiv:2312.01116, 2023.
[69] Y. Lin. Z. Dai, S. Zhu, and Y. Yao, "Gaussian-llow: 4d reconstruction with dynamic 3d gaussian particle." arXiv preprint arXiv:2312.0 22322103311114 2023.
[70] Z. Li, Z. Chen, Z. Li, and Y. Xu, "Spacctime gaussian feature splatting for real-time dynamic view synthesis," arXiv preprint arXiv:2312. 2023.
[71] Y. Liu, H. Guan, C. Luo, L. Fan, J. Peng, and Z. Zhang, "Citygaussian: Real-time high-quality large-scale scene rendering with gaussians," arXiv preprint 4xXiv:2404.01 2024.
[72] B. Kerbl, A. Meuleman, G. Kopanas. M. Wimmer, A. Lanvin, and G. Drettakis, "A hierarchical 3d gaussian representation for real-time rendering of very large datasets." " ACM Transactions on Graphics, vol. 44, no. 3, 2024.
[73] Y. Yan, H. Lin, C. Zhou, W. Wang, H. Sun, K. Zhan, X. Lang, X. Zhou, and S. Peng, "Strect gaussians for modeling dynamic urban scenes," " arXiv preprint arXiv:2401.01 2024.
[74] H. Zhou, J. Shao, L. Xu, D. Bai, W. Qiu, B, Liu, Y. Wang, A. Geiger, and Y. Liao, "Hugs: Holistic urban 3d scene understanding via gaussian splatting." arXiv preprint arxiv:2403.12722. 2024.
[75] X. Zhou, Z. Lin. X. Shan, Y. Wang, D. Sun, and M.-H. Yang. "Drivinggaussian Composite gaussian splatting for surrounding dynamic autonomous driving scenes." arXiv preprint arXiv:2312.07920. 2023.
13

---

[76] J. Yang. B. Ivanovic, O. Litany, X. Weng, S. W. Kim, B. Li, T. Che. D. Xu, S. Fidler, M. Pavone. et al., "Emerner:: Emergent spatial-temporal scene decomposition via sell-supervision" arXiv preprint urXiv:2311.00077 2023.
[77] A. Tonderski, C. Lindström, G. Hess, W. Ljungbergh, L. Svensson, and C. Petersson, "Neurad: Neural rendering for autonomous driving," in CVPR, 2024.
[78] E. J. Stollmitz, T. D. DeRose, and D. H. Salesin, Wavelets for computer graphics: theory and applications. Morgan Kaufmann, 1996.
[79] Z. Wang, C. Bovik, R. Sheikh, and Simoncelli, "Image quality assessment: from error visibility
to structural similarity," IEEE transactions on image processing, vol. 13, no. 4, pp. 600-612 2004.
[80] A. Paszke, S. Gross, F. Massa, Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L.. Antiga, al., "Pytorch: imperative style, high-performance deep learning library," NeurIPS, 2019.
[81] B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al., "Argoverse 2: Next generation datasets for self-driving pereeption and forceasting," arXiv preprint arXiv:2301.00495 2023.
[82] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. "The unreasonable effectiveness of deep features as perceptual metrie," in CVPR, 2018.
[83] Z. Wu, Liu, Luo, Z. Zhong. J. Chen, Xiao, Hou, Lou, Y. Chen, R. Yang, al., "Mars: instance-aware, modular and realistic simulator for autonomous driving." in CAAI International Conference Artificial Intelligence, pp. 3-15, Springer, 2023.
[84] Li, P. Zhou, J.-W. Liu, J. Keppo, Lin, Yan, and Xu, "Instant3d: Instant text-to-3d generation." " arXiv preprint arXiv:2311 08403, 2023.
[85] M. Tancik, E. Weber, E. Ng. Li. B. Yi, T. Wang. A. Kristoffersen. J. Austin, K. Salahi, A. Ahuja, al., "Nerfstudio: modular framework neural radiance field development," ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-12, 2023.
[86] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." arXiv preprint 1rX2v:1412.6680,
2014.
[87] C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey, "Barf: Bundle-adjusting neural radiance fields," in ICCV, 2021.
[88] 7. Pang, 7. Li, and Wang, "Simpletrack: Understanding and rethinking multi-ohject tracking," arXiv preprint 110:211 2021.
14

---

A
Appendix
We provide further details on our method and the experimental setting, as well as additional experi- mental results. We accompany this supplemental material with a demonstration video.
A.1 Demonstration Video
We showcase the robustness of our method by rendering complete free-form trajectory across five highly diverse sequences using the same model. Specifically, we chose the model trained on the residential split in Argoverse 2 [81).
To obtain the trajectory, we interpolate keyframes selected throughout the total geographic area of the residential split into single, smooth trajectory that encompasses most of its spatial extent. We also apply periodical translations and rotations to this trajectory to increase the variety of synthesized viewpoints. We use constant speed of 10 meters per second. We choose five different sequences in the data split as the references, spanning sunny daylight conditions in summer to near sunset in winter. Consequently, the appearance of the sequences changes drastically, e.g. from green, fully-leafed trees to empty branches and snow or from bright sunlight to dark clouds. Furthermore, we render each sequence with its unique set of dynamic objects, simulating various distinct traffic scenarios.
We show that our model is able to perform dynamic view synthesis in all of these conditions at high quality, faithfully representing scene appearance, transient geometry, and dynamic objects in each of the conditions. We highlight that this scenario is extremely difficult, as it requires the model to generalize well beyond the training trajectories, represent totally different appearances and geometry, and model hundreds of dynamic, fast-moving objects. Despite this fact, our method produces realistic renderings, showing its potential for real-world applications.
A.2 Method
Neural field architectures. To maximize efficiency, model and 1 with hash grids and tiny The hash grids interpolate feature vectors at the nearest voxel vertices at multiple levels. The feature vectors are obtained by indexing feature table with hash function. Both neural fields are given input conditioning signals w E R04 and wa = [wo E N. (t) E R 13] and output a color c among the other outputs defined in Section 3.2
MLPs 53
For Φ, we use the Gaussian mean M4 to query the hash function at a certain 3D position yielding an intermediate feature representation fg. We input the feature fg. the sequence latent code w and the base opacity 0g into MLP,, which outputs the opacity attenuation v In parallel branch, input fg. w/. and the viewing direction d encoded by spherical harmonics encoding of degree 4 into the color head MLP. of & that will define the final color of the 3D Gaussian.
For vi, we use a 4D hash function while using only three dimensions for interpolation of the feature vectors, effectively modeling 4D hash grid. We use both the position μ4 and the object code wa. i.e. the object identity, as the fourth dimension of the hash grid to model an arbitrarily large number of objects with a single hash table [6 without a linear increase in memory.
We input the intermediate feature f. and the time encoding 7() into the deformation head MLP, x which will output the non-rigid deformation of the object time if applicable. In parallel, input wi. fgz Yt), and the encoded relative viewing direction d into the color head ML.P a to output the final color. Note that relative viewing direction refers to the viewing direction in canonical, object-centric space. As noted in Section 3.2 the MLP heads are shared across all objects.
We list a detailed overview of the architectures in Table 7 Note that, i) we decrease the hash table experiments to 15 as we find this to be sufficient, and ii) we use two identical networks for to separate rigid from non-rigid object instances.
size of & in single-sequent
Color prediction. The kernel function B& prevents a full saturation the rendered color within the support of the primitive as long the primitive's RGB color is bounded in the [0. 1 range. This can be a problem for background and other uniformly textured regions that contain large 3D Gaussians, specifically larger than single pixel. Therefore, inspired by [84), use a scaled sigmoid activation
15

---

Hash Table
MLP.
MLP.
MLP x
Size
levels
max. res.
layers
neurons
layers
# neurons
layers
neurons
2
2048
2
64
2
8
1024
2
64
2
64
Table Neural field architectures. We provide the detailed parameter configurations of the neural fields we use to represent scene object appearances.
function for the color head MLPo:
1
f(x) = - -iggmoi((cz)
(11)
where e := 0.9 is constant scaling factor. This allows the color prediction to slightly exceed the
valid [0, 1] RGB color space. After alpha compositing, we clamp the rendered RGB to the valid [o, 1] range following (18).
Time-dependent appearance. In addition conditioning the object appearance on the sequence at hand, we model the appearance of dynamic objects as a function of time by inputting 7(t) to MLP as described above. This way, our method adapts to changes scene lighting that are more intricate than the general scene appearance. This could be specular reflections, dynamic indicators such as brake lights, or shadows cast onto the object moves through the environment.
Space contraction. We use space contraction to query unbounded 3D Gaussian locations from the the following function for space contraction:
neural fields (54). In particular,
x, (22
x |
S
1
{(x) =
(12)
X > 1
x
x
For á, we use || I/ as the norm to contract the space, while for 1/ we use the Frobenius norm space contraction for 1/ because 3D Gaussians may extend beyond the 3D) object dimensions to represent e.g. shadows, however, most of the representation capacity should be allocated to the object itself.
|| Il--- Note that
Continuous-tine object poses. Both Argoverse [8 and Waymo Open 23 provide precise timing information for both the LiDAR pointclouds to which the 3D bounding boxes are synchronized, and the camera images. Thus, we treat the dynamic object poses {E ........ <t" } as a continuous function of time (t). Í.e. we interpolate between at to S f < to time t compute (t). This also allows us to render videos at arbitrary frame rates with realistic, smooth object trajectories.
Anti-aliased rendering. Inspired by 59 we compensate for the screen space dilation introduced in 18 when evaluating gf multiplying by compensation factor:
1 1-1(0=
exp
(x) (E
b
(p (pp-(())
(13)
E
bI
where b is chosen to cover a single pixel in screen space. This helps us to render views at different sampling rates.
Gradients of camera parameters. Different from [8], we not only optimize the scene geometry but also the parameters of the camera poses. This greatly improves view quality in scenarios with imperfect camera calibration which frequently the case in street scene datasets. particular, we approximate the gradients w.r.t. camera pose [R|t] as:
OL Øt
Σ
OL 0p
Oc ÖR
Σ
OL du
(1t
T) R.
(14)
This formulation was concurrently proposed in (61. refer to them for detailed derivation. We obtain the gradients w.r.t. the vehicle poses via automatic differentiation [80].
Adaptive density control. We elaborate the modifications described in Section 3.4 Specifically, we observe that the same 3D Gaussian will be rendered at varying dominantly small scales. This biases the distribution of positional gradients towards views where the object is relatively small in
16

---

Vanilla ADC
Modified ADC
Ground Truth
Figure 5: Qualitative comparison of ADCs. We show an example of a close-up car and observe over-smoothing when using vanilla ADC while our modified ADC leads to sharper rendering result.
view space, leading to blurry renderings for close-ups due to insufficient densification. This motivates us to use maximum 2D screen size as an additional splitting criterion.
In addition to the adjustments described above and inspired by recent findings [60], we adapt the criterion of densification during ADC. particular, Kerbl et al. [18] use the average absolute value of
positional gradient 8C across multiple iterations. The positional gradient of projected 3D Gaussian
āμي. the sum of the positional gradients across the pixels covers:
dc
E OL 0p:
(15)
3p, Öp, 0ps
However, this criterion is suboptimal when 3D Gaussian spans more than single pixel, scenario that is particularly relevant for large-scale urban scenes. Specifically, since the positional gradient is composed of sum of per-pixel gradients, these can point in different directions and thus cancel each other out. Therefore, we threshold
Σ OL 0p:
(16)
0p: 8pp
as the criterion to drive densification decisions. This ensures that the overall magnitude of the gradients is considered, independent of the direction. However, this leads to an increased expected value, and therefore increase the densification threshold to 0.0006.
Hyperparameters. We describe the hyperparameters used for our method, while training details can be found in Appendix(A.3 [A.] For ADC, we use an opacity threshold of 0.005 to cull transparent 3D Gaussians. To maximize view quality, we do not cull 3D Gaussians after densification stops. We use near clip plane at 1.0m distance, scaled by the global scene scaling factor. We set this threshold to avoid numerical instability in the projection scales inversely with the depth of the primitive, which causes numerical instabilities as the depth of 3D Gaussian approaches zero. For (t), we use frequencies to encode time l.
3D Gaussians. Indeed, the Jacobian J used in g
A.3 Experiments
Data preprocessing. For each dataset, we obtain the initialization of the 3D Gaussians from point cloud of the scene obtained from the provided LiDAR measurements. To avoid redundant points slowing down training, we voxelize this initial pointcloud with voxel sizes of == 0.1m and T := 0.15m for the single- and multi seuuence experiments, respectively. We use the provided 3D bounding box annotations to filter points belonging to dynamic objects, to initialize the 3D Gaussians for each object, and as our object poses 5.
We use the full resolution 375 x 1242 images for training and evaluation and evaluate at varying training set fractions. For Argoverse 2, we follow the experimental setup of [17]. In particular, we use the full resolution 1550 x 2080 images for training and evaluation and use all cameras of every 10th temporal frame as the testing split. Note that we used the provided masks from [17] to mask out parts of the ego-vehicle for both training and evaluation. For Waymo Open, we follow the experimental setup of EmerNeRF 76). We use the three front cameras (FRONT, FRONT_LEFF
For KITTI and VKITTI, we follow the established benchmark used in [6 83 17 73
FRONT_ RIGHT) and resize the images to 640 960 for both training and evaluation. We use only the first LiDAR return as initial points for our reconstruction. We follow (76) and evaluate the cameras of every 10th temporal frame.
17

---

App. only
Full model
App. only
Full model
RGB
Depth
RGB
Depth
Figure 6: Qualitative examples of transient geometry. We show four relevant examples from the residential split of Argoverse 2 [81]. We observe a large disparity between our full model and ours without transient geometry modeling (App. only). Transient objects like a banner (left bottom) are completely missing and there severe depth and color artifacts (e.g. trees). Best viewed digitally.
For separate evaluation of dynamic objects, compute masks from the 2D ground truth camera bounding boxes. We keep only objects exceeding a velocity m/s to filter for potential sensor and annotation noise. We determine the velocities from the corresponding 3D bounding box annotations. Note also that (76) do not undistort the input images, and we follow this setup for fair comparison.
only the LiDAR measurements at the time of the camera sensor recording as ground truth to ensure dynamic objects receive valid depth supervision. We implement our method in PyTorch [80] with tools from nerfstudio (85). For visualization of the depth, we use the inferno I colormap and linear scaling in the 1-82.5 meters range.
Implementation details. For Cdep.
During training, we use the Adam optimizer [6) with 31 == 0.9, B2 = 0.999. We use separate learning rates for each 3D Gaussian attribute, the neural fields, and the sequence latent codes w In particular, for means J4, we use exponential decay learning rate schedule from 1.6 10-5 to 1.6 10-6 for opacity a, we use a learning rate learning rate of 10 The neural fields are trained with an exponential decay learning rate schedule from 2.5 10-3 to 2.5 10-1 The sequence latent vectors wf are optimized with a learning rate of 5 10-4. 4, We optimize camera and object pose parameters with exponential decay learning rate schedule from 10 to 10 10-6 To counter pose drift, we apply weight decay with factor 10 Note that we also optimize the height of object poses &. We follow previous works [87 g5 [7] and optimize the evaluation camera poses when optimizing training poses to compensate for pose errors introduced by drifting geometry through optimized training poses that may contaminate the view synthesis quality measurement.
10-2, for scales a and rotations q, we use
In our 40GB GPUs for 125,000 steps, taking approximately 2.5 days. our single-sequence experiments, we train our model on single RTX 4090 GPU for several hours. On Waymo Open, we train our model for 60,000 steps while for KITTI and VKITTI2 we train the model for 30,000 steps. For
experiments in Table I and Table 5 we train our model on 8 NVIDIA A100
18

---

Split
3D Box Type
PSNR f
SSIM
LPIPS
Full Image
Dynamic-Only
GT 28.81
0.845
0.260
MLPx
PSNR
SSIM 1
PSNR 1
SSIM 1
Single Seq.
Prediction
28.52
0.842
0.264
29.52
0.891
30.08
0.895
GT 25.78
0.772
0.405
29.60
0
30.15
0.896
Multi. Seq.
Prediction
25.67
0.772
0.409
(b) Ablation on deformation head X. We compare results with and without modeling non-rigid motion as deformations.
(a) Ablation on 3D bounding boxes. We use 3D box annota- tions and predictions from an off-the-shelf 3D tracker.
Table 8: Addttonal ablation studies. In (a) we show results single sequence of the residential area in our henchmark and the full residential area. (h) we use subset of the Dynamic-32 split from (76). Le. the 12 sequences with the highest number of non-rigid objects.
Haniggram 3D Gaussian scales
Eynomic
basckiee
4F
19 ge pale n011900000
(Og boak!
107
12.
10-11
301
0.0
0.5
10
Nean 30 Graan 3cals Rog 200
Skmmber poerre
Figure 7: Runtime comparison of neural fields vs. spherical harmonics. We compare the runtime of querying neural fields & and 1 to a spherical harmonies function of degree 3. We report time- per-query nanoseconds.
Figure 8: Histogram of mean 3D Gaussian scales. We use our model trained on Argoverse (residen- tial split). Both axes are in logarithmic scale. The vast majority of 3D Gaussians have small scale, while there few outliers with huge scales. The scene is approximately within [-1, 1j.
our single-sequence experiments in Table we use a schedule of 100,000 steps. We chose a longer schedule for Waymo Open and Argoverse 2 since the scenes are more complex and contain about 5 - 10x as many images as the sequences in KITTI and VKITTI2.
We linearly scale the number warm-up steps, the steps per ADC, and the maximum step to invoke ADC with the number of training steps. For multi-GPU training, we reduce these parameters linearly with the number of GPUs. However, we observed that scaling the learning rates linearly does perform subpar to the initial learning rates in the multi-GPU setup, and therefore we keep the learning rates the same across all experiments.
Additional ablation studies. In Table Ba, show that while approach benefits from high- quality 3D bounding boxes, it is robust to noise and achieves a high view synthesis quality even with noisy predictions acquired from 3D tracking algorithm [88). In Table Bb, we demonstrate that the deformation head yields a small, albeit noticeable improvement in quantitative rendering results. This
corroborates the utility of deformation head x beyond the qualitative examples shown in Figures 4 and 10 Note that the threshold to distinguish between dynamic and static areas is m following [76] so that some instances like slow-moving pedestrians will be classified as static. Also, since non-rigid entities usually cover only small portion of the scene, expected improvements are inherently small.
In Table fied ADC increases view quality in general, and perceptual quality in par- ticular as it avoids blurry close-up ren- derings. Note that our ADC leads to roughly twice the number of 3D Gaus- sians belonging to objects compared to vanilla ADC, thus avoiding insufficient densification. We also show a qualitative example in Figure illustrating this effect. The close-up car rendering is significantly sharper using the modified ADC. Note that for both variants, we use the absolute gradient criterion (see Appendix A.2) for fair comparison.
show that our modi-
Mod. ADC PSNR
SSIM
LPIPS
Num. Total
Num. Obj.
28.60
0.834
0.270
3.72M
314K
28,81
0.845
0.260
4.IM
G1IK
Table 9: Ablation study on ADC. We compare the performance of vanilla ADC to our modified variant in the single-sequence setting.
19

---

Additional runtime and model analysis. In Figure we provide a comparison in terms of time per query of neural fields of different sizes (equivalent to the sizes of our & and 4) versus querying spherical harmonics function of degree 3 as used in 3DGS [18]. We observe that the SH function approx. to 8x faster evaluate than the neural fields. However, we show that this does not lead to critical increase in overall runtime in Table [ Additionally, we note that the SH function is limited in representation capacity: It not capable of handling varying appearance across input sequences (weather, time of day, season), transient geometry (construction sites, tree leaves), articulated motion of dynamic objects (pedestrians, cyclists), and large-scale scenes with several millions of 3D Gaussians due to memory constraints. Thus, emphasize that the of neural fields does not merely improve memory footprint, but enables applications that 3DGS [8] is not capable of modeling.
In Figure & provide an analysis on the distribution of 3D Gaussian scales across a large-scale urban scene reconstruction. We notice that while the vast majority of 3D Gaussians is small (< 0.001 mean scale), there a small number of very large 3D Gaussians (> 1.0 mean scale) compared to the scene bounds that are approximately within [-1.0, 1.0]. This can lead fog-like artifacts in free- viewpoint renderings. We note that this can mitigated using a regularization term on the 3D Gaussian scales, however, we did not observe quantitative improvement in view synthesis metrics and thus we did not include in the experiments.
Qualitative results. We provide an additional qualitative comparison of the variants iii) and ii) introduced in Section i.e. our model with and without transient geometry modeling. In Figure O we show multiple examples confirming that iii) indeed models transient geometry such as tree leaves or temporary advertisement banners (bottom left), and effectively mitigates the severe artifacts present in the RGB renderings of ii). Furthermore, the depth maps show that iii) faithfully represents the true geometry, while ii) lacks geometric variability across sequences.
In addition, we show qualitative comparisons to the state-of-the-ar in Figure 9 Our method continues produce sharper renderings than the previous pest-performing method 17 while also handling articulated objects such as pedestrians which are missing in the reconstruction of previous works (bottom two rows). Finally, we show another temporal sequence of evaluation frames in Figure I0] Our method handles unconstrained motions and can also reconstruct more complicated scenarios such as pedestrian carrying a stroller (right), or grocery bag (left).
20

---

SUDS T6
MI-NSG 17
4DGF (Ours)
Ground Truth
Figure 9: Additional qualitative results on Argoverse 2 81). We show four examples where the upper two are from the residential area and the lower two from the downtown area.
21

---

YOCA
STOP
JOYA LO
STOP
STOP
STOP
STOP
We show a sequence evaluation views synthesized by our model (top to hottom). We see two pedestrians on the left and right being faithfully modeled across varying body poses while also carrying objects such as stroller (right) or shopping bag (left).
Figure 10: Additional qualitative results on Waymo Open 23
22

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and precede the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], [No] or [NA]. .
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors in their evaluation.
While "[Yes] " is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] "provided proper justification is given (c.g., "error bars are not reported because it would be too computationally expensive" or "we were unable to find the license for the dataset we used"). "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can be found.
general, answering
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only the provided macros for your answers.
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly state the tackled problem, the technical gap in existing work, describe our contributions, and outline experimental results. These claims are in line with our experimental results presented.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
23

---

Justification:
We discuss the limitations of our method in the conclusion.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "I "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sele--ppeiiiainnn. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
Justification: Our paper presents a new system for novel view synthesis and thus relies on experimental validation of our claims.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide extensive implementation details, the hyperparameters, training details of our method, as well as data preprocessing and evaluation protocols the technical
appendix.
24

---

Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We released the full source code for reproducing our experiments. All datasets used are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ctiips.cc/puulic/guui
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
25

---

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See above, we provide all details on data preprocessing, evaluation splits, and others.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Not feasible with the limited computational budget.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by enror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the number and type GPUs as well as the wall clock time to run the experiments.
Guidelines:
26

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics tittps://neurr:
Answer: [Yes]
Justification:
We provide details for reproducing results, we respect all data privacy regula- tions, do not conduct research involving human subjects and discuss the broader impact of our work in our paper.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss this throughout the paper and dedicate paragraph in the conclusion
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
27

---

Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The datasets and benchmarks we use are properly licensed and not scraped but collected by the authors. The datasets respect privacy regulations. Our method does not use data-driven priors derived from, e.g., LLMs.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the papers of each benchmark when using any assets.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release any new assets in our paper.
Guidelines:
The answer NA means that the paper does release assets. Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
28

---

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We did not conduct research involving human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: See above.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "qdczzHYDLg", "reviewer": "Reviewer_f8M6", "summary": "This paper works on novel view synthesis (NVS) for large-scale, dynamic urban scenes. This paper proposes a neural scene representation called 4DGF, which uses 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. The proposed method integrates scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. The method significantly outperforms baselines in terms of speed and rendering quality on three benchmarks.", "questions": "Minor questions/suggestions:\n\nWhat does GPU memory in Tab. 4 (a) mean? Is it peak memory?\n\nIn Fig. 1 inputs, the blue/orange colors for the image boundaries are also used for \"geometry\" and \"appearance\" respectively. I assume there's not such a correspondence between the input and geometry/appearance. So maybe you could change the input boundaries to different colors.\n\nFIg. 2: extra space in \"Scene configuration\"\n\nTab. 4 appears before Tab. 3, maybe you could switch the order.", "limitations": "The limitations seem to have been sufficiently discussed.", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The idea of combining Gaussian Splatting and neural fields to model geometry and appearance, respectively, is very interesting. This makes a lot of sense considering the efficiency and the advantages of each of the two representation. This is definitely a more scalable approach to large-scale scenes compared to prior work.\n\n2. Extensive experiments have been conducted to validate the proposed method, this includes comparing with recent baselines on three benchmarks and the ablation studies that carefully examine each component. Moreover, the rendering quality improvement and the speedup is very significant on all three datasets.\n\n3. The paper is very well-written and easy to follow. Implementation details are sufficiently discussed for reproducibility.", "weaknesses": "1. I appreciate the authors' including a video in the submission. I found sometimes there's a large foggy region near the camera (e.g., the regions on the right during the 5-6th second), do the authors have any explanations on that? Is it caused by any limitations discussed in Sec. 5?\n\n2. I understand that this paper mainly focuses on large dynamic scenes. I am curious how this hybrid representation performs on 3D statics scenes (e.g., the benchmarks that the original 3DGS have been tested on). This seems to be a more straightforward way to see the effect of using neural fields instead to model appearance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our manuscript and the constructive feedback on our work. Below we address the concerns raised.\n1. Foggy regions in the demonstration video: While these artifacts may be caused by the limitations discussed in Sec. 5 like white balance or focus blur, we suspect it is due to an issue with 3DGS optimization. In particular, we noticed that 3DGS tends to produce a small number of very large, semi-transparent 3D Gaussians, especially when pruning thresholds are set generously. In Fig. 1 of the PDF attached to the global response, we show there are a few 3D Gaussians with a huge mean scale (>1.0), while the vast majority is small (<0.001 mean scale) compared to the scene bounds that are approximately within [-1.0, 1.0]. Concurrently proposed techniques like scale regularization [Kheradmand et al.] could mitigate this issue.\n2. Experiments on static 3D scenes: In Tab. 4a, we show that our method performs similarly when using neural fields as color representation compared to the SH representation from 3DGS [18] (row 2 vs. 3). This illustrates that, on single-sequence, homogeneous data, SH and neural fields are equivalently suitable for modeling appearance, both for static and dynamic scene parts. This finding is in line with concurrent work focusing on static 3D scenes [Lu et al.]. In contrast, Tab. 4b shows that, on multi-sequence, heterogeneous data, modeling appearance and transient geometry with neural fields is essential for NVS performance. However, we agree this is an interesting experiment and will add this comparison to the camera-ready version.\n3. GPU memory in Tab. 4a: We report peak memory consumption as it is the deciding factor when attempting to train a 3DGS model on a specific scene.\n4. Suggestions on Fig. 1/2, Tab. 3/4: Thank you for spotting these issues. We will correct them in the camera-ready version.\n\nWe hope our response adequately addresses the reviewer’s concerns. We are happy to provide more information during the discussion phase.\n\n[Kheradmand et al.] 3D Gaussian Splatting as Markov Chain Monte Carlo\n\n[Lu et al.] Scaffold-GS: Structured 3D Gaussians for View-Adaptive Rendering", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a1HKXkD7lO", "reviewer": "Reviewer_6enk", "summary": "The paper presents a novel 3D scene representation for novel view synthesis (nvs) in dynamic urban environments where, in particular, under heterogeneous imaging environments. The proposed representation relies on existing ingredients: 3D Gaussian Splatting, learned static/dynamic object instances, and  a global scene graph.\n\nThe resulting system yields very strong results on a series of public autonomous driving benchmarks.", "questions": "I do not have more questions or suggestions than the ones underlying the aforementioned weaknesses.", "limitations": "The authors provide one dedicated paragraph that reasonably addresses such considerations.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "### + Readability.\nOverall, in its current state, the paper's readability is relatively good. The main ideas, concepts, are mostly well discussed, conveyed, and articulated, throughout the paper.\n\n### + Practical usefullness of the considered problem.\n\n### + Structure, and Organization of the Contents.\nThe presentation is mostly on point and each dedicated section of the paper is properly balanced. The use of text real-estate is fair.\n\n### + Relative simplicity of the conceptual contribution.\n\n### + The amount of implementation details is very good.\n\n### + The reported performance.\n\n### + Implementation details for reproducibility: excellent.", "weaknesses": "### - (1) Positioning of the conceptual contribution vs. the competitive landscape.\n\nIn particular, the proposed method looks very much like a revisit of Panoptic Radiance Fields [49] by replacing the NeRF component byt 3D Gaussian splats. \n\nWhile this is perfectly fine, this merits a targeted, transparent discussion in the main paper to help the reader understand the whereabouts of how the proposed contribution relates (or not) with such pieces of litterature.\n\n### - (2) How much does it cost?\n\nMissing piece of information regarding the resource usage, memory footprint, typical timings etc to better understand the downsides of using the provided method.\n\n### - (3) (To a lesser extent) Certain contents in the paper are unclear.\n\nFigure 4: what is happening? Adding color annotations or boxes would definitely help.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback and for taking the time to review our manuscript. Below we address the concerns raised.\n1. Positioning of the conceptual contribution vs. the competitive landscape: We addressed this in  L114-117 of our paper, but we will make the distinction clearer in the camera-ready version. Specifically, while our work shares similarities to [49] in how the rigid, global motion of dynamic objects is handled, our method fundamentally differs in several key aspects: 1. We introduce a mechanism to handle both transient geometry and varying appearance across multiple, *heterogeneous* captures, a key component that [49] is not addressing 2. This mechanism and the efficiency of our approach enable the reconstruction of much larger urban areas with significantly more dynamic objects than in [49], 3. While [49] excludes non-rigid objects, we model the articulated motion of non-rigid dynamic objects, and 4. We do not rely on semantic priors to represent dynamic objects accurately.\n2. Runtime/memory cost: We address it in our global response. In short, we show that the neural fields introduced by our method do not dominate the runtime. We report peak memory consumption in Tab. 4a and show that our method leads to a reduced memory footprint compared to the SH color representation in 3DGS [18].\n3. Figure 4: Thank you for pointing out this issue. The important bit is happening in the middle of the image: The lady in the pink sweater is getting out of the dark grey car in front of the ego-vehicle. It illustrates the ability of our method to handle complex, articulated motions. We will add colored boxes to the camera-ready version to make it clear.\n\nWe hope our response adequately addresses the reviewer’s concerns. We are happy to provide more information during the discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVm4uQmqVd", "reviewer": "Reviewer_7pd2", "summary": "This paper proposes a hybrid neural scene representation for dynamic urban driving scene modelling. The method utilizes 3D Gaussians as an efficient geometric scaffold and neural fields to represent appearance, thereby reducing memory. To account for transient scene geometry variations caused by weather, seasons, and other factors, the authors introduce an opacity attenuation field that modulates the scene geometry. For modeling dynamic actors in the scene, an object-centric representation is used, with a non-rigid deformation in the canonical space to animate objects such as pedestrians. Experiments demonstrate that the proposed method achieves state-of-the-art performance while rendering faster than previous methods.", "questions": "This paper addresses a practical and important problem in autonomous driving. The writing is clear, and the results are promising. I look forward to the authors' response to the concerns I raised above.", "limitations": "The authors discussed the limitations of modeling other sensor phenomena such as rolling shutter effects, motion, and more complex camera models. They also discussed the broader societal impacts of their work.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper is well-written and easy to follow.\n* The decomposed representation of appearance significantly reduces memory usage.\n* It models transient scene appearance and geometry, as well as non-rigid objects like pedestrians.\n* The evaluation and ablation study are comprehensive.\n* The paper demonstrates visually superior results compared to baselines such as SUDS and ML-NSG.", "weaknesses": "* The rendering of the proposed scene representation requires query appearance from the neural fields, it is unclear whether this will impact rendering speed compared to spherical harmonics representation.\n* This paper lacks a comparison with recent neural field baselines such as UniSim and NeuRAD for urban driving scenes. Additionally, there is no comparison of the speed to 3D Gaussian baselines.\n* How to control the non-rigid objects in the scene? e.g., animating the pedestrians given a sequence of human poses.\n* Is it feasible to render other sensor modalities in autonomous driving, such as LiDAR?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for taking the time to review our manuscript and the constructive feedback on our work. Below we address the concerns raised.\n1. Impact of neural field query on rendering speed: We address this point in the global rebuttal posted above. In a nutshell, we show that the neural field query accounts for only 13% of the total runtime, so while the neural field query is slower compared to spherical harmonics, the impact on the rendering speed is marginal.\n2. Additional comparisons: We thank the reviewer for pointing us to UniSim and NeuRAD. While at submission time, we were not aware that the code for either of the two papers was public, we noticed that the code for NeuRAD was released by now. Therefore, we provide an additional comparison on KITTI MOT in Tab. 2 of the PDF attached to the global response and show that our method outperforms NeuRAD. Regarding further comparison to 3D Gaussian baselines, we refer the reviewer to Tables 2 and 4a of our paper, where we compare to StreetGaussians [73] and the SH color representation used in 3DGS [18] (row 2 vs. 3), respectively.\n3. Control of non-rigid objects: We build a model capable of coping with general dynamic urban scenarios, which exhibit a wide range of dynamic actors. Thus, the design of our method is such that it allows for general modeling of non-rigid objects, e.g. pedestrians holding a stroller or shopping bags (see Fig. 8 in our supplemental material for an example), cyclists, and animals. Specifically, the articulated motion of pedestrians is controlled via time input to the deformation head, not via human pose estimates. \n4. LiDAR rendering: While we did not consider rendering LiDAR measurements in our work, it is possible with our representation. However, we note that the 3DGS renderer assumes a pinhole camera model, which differs from the LiDAR sensor model. Therefore, ray-tracing-based approaches for rendering 3D Gaussians should be utilized [Yu et al.]. In addition, the LiDAR intensity and ray drop probability should be modeled with the neural fields following e.g. NeuRAD. The LiDAR point cloud can then be rendered from rays generated by the LiDAR sensor model. We consider this as an interesting direction to explore for future work.\n\nWe hope this adequately addresses the reviewer’s concerns. We are happy to provide more information during the discussion phase.\n\n[Yu et al.] Gaussian Opacity Fields: Efficient and Compact Surface Reconstruction in Unbounded Scenes", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xL9aqsl9tV", "reviewer": "Reviewer_51SN", "summary": "This paper aims to perform view synthesis for dynamic urban scenes. This paper adopts 3DGS as scene geometry and uses neural fields to model the dynamic appearance of urban scenes. The neural scene graph is introduced to handle the movement of dynamic objects, and a deformation field is used to handle local articulated motions. Experiments show that the proposed approach outperforms baseline methods.", "questions": "The presentation of this paper should be improved. Some important technical details are missing. The limitations from the introduction of neural fields should be discussed.", "limitations": "The limitations from the introduction of neural fields should be discussed.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The presented pipeline well handles the dynamic appearance of urban scenes.\n2. The experiments are sufficient and validate the effectiveness of the proposed approach.\n3. The idea of combining neural fields with 3DGS is sound and effective.", "weaknesses": "1. The method presented in the paper takes 0.17 seconds to render an image at a resolution of 1550x2048, which is significantly slower than conventional 3DGS. Is the trade-off of such a significant sacrifice in rendering speed for quality improvement justified? Does the author have any solutions to address this issue?\n2. The paper needs to evaluate the extent to which neural fields impact the rendering speed of 3DGS.\n3. The pipeline figure of the paper should be clearer. The connections between the various modules are not easily discernible from the figure and its caption. For instance, it is not clearly depicted how the latent codes obtained from the scene configuration are inputted into the neural fields. Then, how are neural fields combined with 3DGS to represent static scenes and dynamic objects? The figure only shows simple association arrows. However, these modules are not merely input-output relationships. There are some combination operations between them.\n4. The paper uses neural fields to represent appearance, which reduces the memory footprint but may also significantly impact rendering speed. Has the paper considered how to address this issue?\n5. In Figure 2 of the paper, regarding the neural fields section, the symbols for static opacity correction and dynamic deformation are inconsistent with the descriptions in Section 3.2 of the paper. This is quite confusing.\n6. I am curious whether the combination of neural fields with 3DGS could make the optimization of 3DGS unstable?\n7. The non-rigid objects mentioned in the paper refer to cars, right? Or other objects? I did not see how the paper describes the modeling of cars. Although the paper mentions the use of scene graphs for modeling, I did not see how dynamic cars are represented using scene graphs. Does the paper treat dynamic cars as non-rigid objects directly? In this case, how can the large range of movement of dynamic cars be handled?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful feedback and for taking the time to review our manuscript. Below we address the concerns raised.\n1. Rendering speed compared to 3DGS: We hope to address this concern in our global response, where we show a) an improved runtime of 0.074 seconds when rendering frames at 1550x2048 resolution using 8.02M 3D Gaussians, and b) a detailed runtime analysis of our method. This analysis reveals that the rasterization and the scene graph evaluation necessary for rendering dynamic scenes account for more than 75% of the total runtime. While we acknowledge that rendering large-scale urban dynamic scenes is more complex than rendering small-scale static 3D scenes and thus the reported rendering speeds are significantly slower than in 3DGS [18], we believe that the complex nature of the scenes we render needs to be taken into consideration when examining these numbers.\n2. Figure 2 unclear: We plan to improve the clarity of the figure for the camera-ready version, highlighting the information flow. To render an image at sequence $s$ and time $t$, we first evaluate the scene graph which returns the latent codes (i.e. the conditioning signals for the neural fields), as well as coordinate transformations and the active set of 3D Gaussians (i.e. which objects are present where). We then use the latent codes in conjunction with the sets of 3D Gaussians as input to the neural fields to retrieve the 3D Gaussian color and the opacity correction or deformation. After this, we use the coordinate transformations to compose the scene globally, i.e. placing the objects at the right locations. Finally, we render the scene with the 3DGS renderer.\n3. Impact of neural fields on rendering speed: In our global response above, we show that querying the neural fields accounts for only 13% of the total rendering speed. This is because we use efficient hash-grid based neural fields inspired by [53]. Therefore, although the neural fields are slower to query than the spherical harmonics used in 3DGS [18], it does not critically impact rendering speed.\n4. Figure 2 symbols: We thank the reviewer for pointing this out, indeed we abbreviated $\\delta \\mu_k^t$ in the figure to $\\delta$ to make it more concise. We will correct this in the camera-ready version.\n5. Impact of neural fields on optimization: In our experiments, the optimization was not negatively impacted by the introduction of neural fields. In particular, we show in Tab. 4a that replacing the SH color function with neural fields does not lead to a significant change in performance while halving the GPU memory requirements (row 2 vs. 3).\n6. Dynamic vs. non-rigid objects: We apologize for the confusion. In L196-197 of the paper, we refer to vehicles (e.g. cars) as rigid dynamic objects since their motion can be described with a single rigid body transformation, while we refer to pedestrians, cyclists, and others as non-rigid dynamic objects since their motion is articulated, i.e. cannot be represented by a single rigid body transformation. We represent the global motion of all dynamic objects with our scene graph, transforming the objects from an object-centric canonical space to world space with rigid body transformations. Additionally, we represent the local, articulated motion of non-rigid objects like pedestrians as deformation in canonical space. We hope this clarifies the reviewer’s concern and we will adjust our writing in the camera-ready version accordingly.\n\nWe hope that our response adequately addresses the reviewer’s concerns. We are happy to provide more information during the discussion phase.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all reviewers for their helpful and constructive feedback. We appreciate the positive reception of our work. The consensus is that the combination of neural fields and 3D Gaussians as scene representation constitutes an interesting (f8M6), conceptually simple (6enk), and effective (51SN) solution to practical problems with 3DGS [18], i.e. memory consumption and thus scalability to large-scale scenes (f8M6, 7pd2) as well as modeling of complex real-world phenomena such as transient geometry and appearance across captures and non-rigid objects (7pd2). The reviewers point out that the improvements over existing work are convincing with visually superior results compared to prior art (f8M6, 7pd2, 6enk), and acknowledge the comprehensive evaluation and ablation studies (f8M6, 7pd2, 51SN) validating the effectiveness of our approach. Finally, most reviewers agree that the paper is well-written, easy to follow (f8M6, 6enk, 7pd2), and the presentation is mostly on point (6enk). The reviewers also acknowledge the extensive discussion of experimental and implementation details for reproducibility (f8M6, 6enk).

A shared concern among the reviews is that a runtime analysis should be included. We are thankful for this suggestion and a) include a detailed runtime analysis in the attached PDF, and b) *improve the runtime of our algorithm by more than 2x*. In particular, we implement two improvements:  First, we significantly reduce the number of queries to the neural fields by skipping out-of-view 3D Gaussians after projection. Second, we implement a vectorized version of the neural field query that retrieves the outputs for all dynamic objects in parallel. Next, we measure the runtime of each pipeline component and report the results in Tab. 1. It shows that the runtime is dominated by rasterization and scene graph evaluation, accounting for more than 75% of the total average runtime, not by the neural field queries (approx. 13%). This is because we render views at high resolution with a large number of 3D Gaussians which is computationally demanding even for the efficient rasterization of 3DGS [18], and because we represent areas with hundreds to thousands of dynamic objects, making the retrieval of the 3D Gaussians and latent codes costly. For the neural fields, we use very efficient hash-grid based encodings in conjunction with tiny MLPs, inspired by InstantNGP [53]. Thus, simply replacing the neural fields with the SH function of [18] would not lead to a significant speedup. Note also that our method is significantly faster in simpler scenarios with fewer 3D Gaussians per scene, e.g. Waymo Open. 
In addition, we provide a comparison in terms of time per query of the SH function used in 3DGS [18] versus our hash-grid based neural field query. While the SH function is faster to evaluate than the neural fields, it is also severely limited in representation capacity: It is not capable of handling varying appearance across input sequences (weather, time of day, season),  transient geometry (construction sites, tree leaves), articulated motion of dynamic objects (pedestrians, cyclists), and large-scale scenes with several millions of 3D Gaussians due to memory constraints. Thus, we emphasize that the use of neural fields does not merely improve results, but *enables applications that 3DGS [18] is not capable of modeling*. We will dedicate a paragraph to the runtime analysis in the camera-ready version.

Meta-review: The paper proposed a new 3D scene representation that combines the strength of 3DGS and neural field to model dynamic urban scenes, which leads to better efficiency, lower memory consumption, and higher quality on benchmark datasets, while being able to handle the transient geometry and appearance. The decomposition of dynamic and static components further helps to model non-rigid motion, such as pedestrians. These are valuable contributions to our research community.  

The paper received mixed reviews initially, but the rebuttal and following discussions successfully addressed the concerns of all the reviewers who all became positive after the rebuttal. After discussing with SCA, the AC agrees with the reviewers' opinions on accepting this paper.  As promised during the rebuttal and discussion, please add the additional results of running time comparisons and analysis provided during the rebuttal to the main paper.
Decision: Accept (spotlight)