Dynamic 3D Gaussian Fields for Urban Areas

Authors: tobias  fischer
jonas  kulhanek
samuel rota bulò
lorenzo  porzi
marc  pollefeys
peter  kontschieder
eth  zürich
meta reality labs
ctu  prague
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 23:47:28

Abstract: We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200x in rendering speed.

Main paper: 
 introduction: 
 The problem of synthesizing novel views from a set of images has received widespread attention in recent years due to its importance for technologies like AR/VR and robotics. In particular, obtaining interactive, high-quality renderings of large-scale, dynamic urban areas under varying weather, lighting, and seasonal conditions is a key requirement for closed-loop robotic simulation and immersive VR experiences. To achieve this goal, sensor-equipped vehicles act as a frequent data source that is becoming widely available in city-scale mapping and autonomous driving, creating the possibility of building up-to-date digital twins of entire cities. However, modeling these scenarios is extremely challenging as heterogeneous data sources have to be processed and combined: different weather, lighting, seasons, and distinct dynamic and transient objects pose significant challenges to the reconstruction and rendering of dynamic urban areas.In recent years, neural radiance fields have shown great promise in achieving realistic novel view synthesis of static [1, 2, 3] and dynamic scenes [4, 5, 6, 7] . While earlier methods were limited to controlled environments, several recent works have explored large-scale, dynamic areas [8, 9, 10] . Among these, many works resort to removing dynamic regions and thus produce partial reconstructions [9, 10, 11, 12, 13, 14] . In contrast, fewer works model scene dynamics [15, 16, 17] . These methods exhibit clear limitations, such as rendering speed which can be attributed to the high cost of ray traversal in volumetric rendering. Therefore, rasterization-based techniques [18, 19, 20, 11] have recently emerged as a viable alternative. Most notably, Kerbl et al. [18] propose a scene representation based on 3D Gaussian primitives that can be efficiently rendered with a tile-based rasterizer at a high visual quality. While demonstrating impressive rendering speeds, it requires millions of Gaussian primitives with high-dimensional spherical harmonics coefficients as color representation to achieve good view synthesis results. This limits its applicability to large-scale urban areas due to high memory requirements. Furthermore, due to its explicit color representation, it cannot model transient geometry and appearance variations commonly encountered in city-scale mapping and autonomous driving use cases such as seasonal and weather changes. Lastly, the approach is limited to static scenes which complicates representing dynamic objects such as moving vehicles or pedestrians commonly encountered in urban areas.To this end, we propose 4DGF, a method that takes a hybrid approach to modeling dynamic urban areas. In particular, we use 3D Gaussian primitives as an efficient geometry scaffold. However, we do not store appearance as a per-primitive attribute, thus avoiding more than 80% of its memory footprint. Instead, we use fixed-size neural fields as a compact and flexible alternative. This allows us to model drastically different appearances and transient geometry which is essential to reconstructing urban areas from heterogeneous data. Finally, we model scene dynamics with a graph-based representation that maps dynamic objects to canonical space for reconstruction. We model non-rigid deformations in this canonical space with our neural fields to cope with articulated dynamic objects common in urban areas such as pedestrians and cyclists. This decomposed approach further enables a flexible scene composition suitable to downstream applications. The key contributions of this work are:• We introduce 4DGF, a hybrid neural scene representation for dynamic urban areas that leverages 3D Gaussians as an efficient geometry scaffold and neural fields as a compact and flexible appearance representation.• We use neural fields to incorporate scene-specific transient geometry and appearances into the rendering process of 3D Gaussian splatting, overcoming its limitation to static, homogeneous data sources while benefitting from its efficient rendering.• We integrate scene dynamics via i) a graph-based representation, mapping dynamic objects to canonical space, and ii) modeling non-rigid deformations in this canonical space. This enables effective reconstruction of dynamic objects from in-the-wild captures.We show that 4DGF effectively reconstructs large-scale, dynamic urban areas with over ten thousand images, achieves state-of-the-art results across four dynamic outdoor benchmarks [21, 22, 17, 23] , and is more than 200× faster to render than the previous state-of-the-art. 
related work: 
 Dynamic scene representations. Scene representations are a pillar of computer vision and graphics research [24] . Over decades, researchers have studied various static and dynamic scene representations for numerous problem setups [25, 26, 27, 28, 29, 30, 31, 32, 1, 33, 34] . Recently, neural rendering [35] has given rise to a new class of scene representations for photo-realistic image synthesis. While earlier methods in this scope were limited to static scenes [2, 3, 36, 37, 38] , dynamic scene representations have emerged quickly [4] . These scene representations can be broadly classified into implicit and explicit representations. Implicit representations [5, 6, 7, 39, 4, 40, 41, 16] encode the scene as a parametric function modeled as neural network, while explicit representations [42, 43, 44, 45, 44] use a collection of low-level primitives. In both cases, scene dynamics are simulated as i) deformations of a canonical volume [5, 6, 42, 39, 41] , ii) particle-level motion such as scene flow [7, 4, 40, 16, 46] , or iii) rigid transformations of local geometric primitives [44] . On the contrary, traditional computer graphics literature uses scene graphs to compose entities into complex scenes [47] . Therefore, another area of research explores decomposing scenes into higher-level elements [31, 48, 32, 15, 49, 50, 17] , where entities and their spatial relations are expressed as a directed graph. This concept was recently revisited for view synthesis [15, 17] . In this work, we take a hybrid approach that uses i) explicit geometric primitives for fast rendering, ii) implicit neural fields to model appearance and geometry variation, and iii) a scene graph to decompose individual dynamic and static components.Efficient rendering and 3D Gaussian splatting. Aside from accuracy, the rendering speed of a scene representation is equally important. While rendering speed highly depends on the representation efficiency itself, it also varies with the form of rendering that is coupled with it to generate an image [51] . Traditionally, neural radiance fields [3] use implicit functions and volumetric rendering which produce accurate renderings but suffer from costly function evaluation and ray traversal. To remedy these issues, many techniques for caching and efficient sampling [52, 53, 54, 36, 55, 17] have been developed. However, these approaches often suffer from excessive GPU memory requirements [52] and are still limited in rendering speed [54, 55, 17] . Therefore, researchers have opted to exploit more efficient forms of rendering, baking neural scene representations into meshes for efficient rasterization [19, 20, 11] . This area of research has recently been disrupted by 3D Gaussian splatting [18] , which i) represents the scene as a set of anisotropic 3D Gaussian primitives ii) uses an efficient tile-based, differentiable rasterizer, and iii) enables effective optimization by adaptive density control (ADC), which facilitates primitive growth and pruning. This led to a paradigm shift from baking neural scene representations to a more streamlined approach.However, the method of Kerbl. et al. [18] exhibits clear limitations, which has sparked a very active field of research with many concurrent works [44, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65] . For instance, several works tackle dynamic scenes by adapting approaches described in the paragraph above [44, 66, 67, 68, 69, 70] . Another line of work focuses on modeling larger-scale scenes [65, 71, 72] . Lastly, several concurrent works investigate the reconstruction of dynamic street scenes [73, 74, 75] . These methods are generally limited to homogeneous data and in scale. In contrast, our method scales to tens of thousands of images and effectively reconstructs large, dynamic urban areas from heterogeneous data while also providing orders of magnitude faster rendering than traditional approaches.Reconstructing urban areas. Dynamic urban areas are particularly challenging to reconstruct due to the complexity of both the scenes and the capturing process. Hence, significant research efforts have focused on adapting view synthesis approaches from controlled, small-scale environments to larger, real-world scenes. In particular, researchers have investigated the use of depth priors from e.g. LiDAR, providing additional information such as camera exposure, jointly optimizing camera parameters, and developing specialized sky and light modeling approaches [8, 9, 10, 11, 12, 13, 14] . However, since scene dynamics are challenging to approach, many works simply remove dynamic areas, providing only a partial reconstruction. A few works explicitly model scene dynamics, but suffer from limitations in terms of scalability [15, 49, 45] , accuracy [16] , rendering speed [17, 76, 77] , or modeling of non-rigid and uncommon objects [15, 49, 45, 77, 17] . We introduce a mechanism to handle transient geometry and varying appearance, improve rendering efficiency, and, inspired by how global rigid object motion is handled in [17, 15, 49, 45, 77] , propose an approach to model the local articulated motion of non-rigid dynamic objects without using semantic priors. Consequently, our work enables the reconstruction of much larger urban areas with a significantly higher number and diversity of dynamic objects across multiple in-the-wild captures.3 Method 
problem setup: 
 We are given a set of heterogeneous sequences S that capture a common geographic area from a moving vehicle. The vehicle is equipped with calibrated cameras mounted in a surround-view setup.We denote with C s the set of cameras of sequence s ∈ S and with C the total set of cameras, i.e. C := s∈S C s . For each camera c ∈ C, we assume to know the intrinsic K c parameters and the pose P c ∈ SE(3), expressed in the ego-vehicle reference frame. Ego-vehicle poses P t s ∈ SE(3) are provided for each sequence s ∈ S and timesteps t ∈ T s and are expressed in the world reference frame that is shared across all sequences. Here, T s denotes a set of timestamps relative to s. Indeed, we assume that timestamps cannot be compared across sequences because we lack a mapping to a global timeline, which is often the case with benchmark datasets due to privacy reasons. For each sequence s ∈ S, camera c ∈ C s and timestamp t ∈ T s we have an RGB image I t (s,c) ∈ [0, 1] H×W ×3t i ∈ T o ⊂ T s .Our goal is to estimate the plenoptic function for the shared geographic area spanned by the training sequences, i.e. a function f (P, K, t, s), which outputs a rendered RGB image of size (H, W ) for a given camera pose P with calibration K in the conditions of sequence s ∈ S at time t ∈ T s .We model a parameterized, plenoptic function f θ , which depends on the following components: i) a scene graph G that provides the scene configuration and latent conditioning signals ω for each sequence s, object o, and time t, ii) sets of 3D Gaussians that serve as a geometry scaffold for the scene and objects, and iii) implicit neural fields that model appearance and modulate the geometry scaffold according to the conditioning signals. See Figure 2 for an overview of our method. Scene configuration. Inspired by [17] , we factorize the scene with a graph representation G = (V, E), holding latent conditioning signals at the nodes V and coordinate system transformations along the edges E. The nodes V consist of a root node v r defining the global coordinate system, camera nodes {v c } c∈C , and for each sequence s ∈ S, sequence nodes {v t s } t∈Ts and dynamic object nodes {v o } o∈Os . We associate latent vectors ω to sequence and object nodes representing local appearance and geometry. Specifically, we model the time-varying sequence appearance and geometry viaEQUATIONwhere A s and G s are appearance and geometry modulation matrices, respectively, and γ(•) is a 1D basis function of sines and cosines with linearly increasing frequencies at log-scale [35] . Time t is normalized to [-1, 1] via the maximum sequence length max s∈S |T s |. For objects, we use both an object code and a time encodingω t o := [ω o , γ(t)] .(2) Nodes in the graph G are connected by oriented edges that define rigid transformations between the canonical frames of the nodes. We have P t s for sequence to root edges, P c for camera to sequence edges, and ξ t o for object to sequence edges. 3D Gaussians. We represent the scene geometry with sets of anisotropic 3D Gaussians primitives G = {G r } ∪ {G o : o ∈ O s , s ∈ S}. Each 3D Gaussian primitive g k is parameterized by its mean µ k , covariance matrix Σ k , and a base opacity α k . The covariance matrix is decomposed into a rotation matrix represented as a unit quaternion q k and a scaling vector a k ∈ R 3 + . The geometry of g k is represented byEQUATIONThe common scene geometry scaffold is modeled with a single set of 3D Gaussians G r , while we have a separate set G o of 3D Gaussians for each dynamic object o. Indeed, scene geometry is largely consistent across sequences while object geometries are distinct. The 3D Gaussians G r are represented in world frame, while each set G o is represented in a canonical, object-centric coordinate frame, which can be mapped to the world frame by traversing G.Differently from [18] , our 3D Gaussians do not hold any appearance information, reducing the memory footprint of the representation by more than 80%. Instead, we leverage neural fields to regress a color information c s,t k and an updated opacity α s,t k for each sequence s ∈ S and time t ∈ T s . For 3D Gaussians in G r modeling the scene scaffolding, we predict an opacity attenuation term ν s,t k that is used to model transient geometry by downscaling α k . Instead, for 3D Gaussians in G o modeling objects the base opacity is left invariant. HenceEQUATION)The attenuation term enforces a high base opacity for every 3D Gaussian visible in at least one sequence. Therefore, we can obtain pruning decisions in ADC by thresholding the base opacity α k , which is directly accessible without computational overhead, without risking the removal of transient geometry.Furthermore, in the presence of non-rigid objects o, we predict deformation terms δ t k ∈ R 3 to the position of 3D primitives in G o via the neural fields, for each time t ∈ T o . In this case, the position of the primitive in object-centric space at time t is given byEQUATIONAppearance and transient geometry. Given the scene graph G and the 3D Gaussians G, we use two neural fields to decode the aforementioned parameters for each primitive. In particular, for 3D Gaussians in G r modeling the static scene, the neural field is denoted by ϕ and regresses the opacity attenuation term ν s,t k and a color c s,t k , given the 3D Gaussian primitive's position µ k , a viewing direction d, the base opacity α k and the latent code of the node ω t s , i.e.EQUATIONwhere s ∈ S and t ∈ T s . Note that since the opacity attenuation ν Both µ k and d are expressed in the canonical, object-centric space of object o. Using neural fields has three key advantages for our purpose. First, by sharing the parameters of ϕ and ψ across all 3D Gaussians G, we achieve a significantly more compact representation than in [18] when scaling to large-scale urban areas. Second, it allows us to model sequence-dependent appearance and transient geometry which is fundamental to learning a scene representation from heterogeneous data. Third, information sharing between nodes enables an interaction of sequence and object appearance.However, querying a neural field is more complex than a spherical harmonics function as in [18] . Therefore, we i) use efficient hash-grid representations [53] to minimize query complexity and, ii) carefully optimize the rendering workflow to minimize the amount of queries. In particular, we skip out-of-view 3D Gaussians and implement a vectorized query function to ψ that retrieves the parameters of all relevant dynamic objects in parallel. We refer to Section 4.3 for a runtime analysis.Non-rigid objects. Street scenes are occupied not only by rigidly moving vehicles but also by, e.g., pedestrians and cyclists that move in a non-rigid manner. These pose a significant challenge due to their unconstrained motion under limited visual coverage. Therefore, we take a decomposed approach to modeling non-rigid objects. First, we represent the local articulated motion of non-rigid objects like pedestrians as deformation in canonical space. We use deformation head χ that predicts a local position offset δ t k viaEQUATION) given an intermediate feature representation f ψ of ψ conditioned on µ k and time t. We deform the position of µ k over time in canonical space as per Equation ( 5). Second, we use the scene graph G to model the global rigid object motion, transforming the objects from object-centric to world space with a rigid body transformation. We use a general design to cover a wide range of scenarios, such as pedestrians holding a stroller or shopping bags, cyclists, and animals. See Figure 10 in our supp. mat.Background modeling. To achieve a faithful rendering of far-away objects and the sky, it is important to have a background model. Inspired by [54] , where points are sampled along a ray at increasing distance outside the scene bounds, we place 3D Gaussians on spheres around the scene with radius r2 i+1 for i ∈ {1, 2, 3} where r is half of the scene bound diameter. To avoid ambiguity with foreground scene geometry and to increase efficiency, we remove all points that are i) below the ground plane, ii) occluded by foreground scene points, or iii) outside of the view frustum of any training view. To uniformly distribute points on each sphere, we utilize the Fibonacci sphere sampling algorithm [78] , which arranges points in a spiral pattern using a golden ratio-based formula. Even though this sampling is not optimal, it serves as a faster approximation of the optimal sampling.Scene composition. To render our representation from the perspective of camera c at time t in sequence s, we traverse the graph G to obtain the latent vector ω t s and the latent vector ω t o of each visible object o ∈ O s , i.e. such that t ∈ T o . Moreover, for each 3D Gaussian primitive g k in G, we use the collected camera parameters, object scale, and pose information to determine the transformation Π c k mapping points from the primitive's reference frame (e.g. world for G r , object-space for G o ) to the image space of camera c. Opacities α s,t k are computed as per Equation (4), while colors c s,t k are computed for primitives in G r and in G o via Equations ( 6) and (7) , respectively. For non-rigid objects in G o , we compute the primitive positions µ t k via Equation ( 5). Rasterization. To render the scene from camera c, we follow [18] and splat the 3D Gaussians to the image plane. Practically, for each primitive, we compute a 2D Gaussian kernel denoted by g c k with mean µ c k given by the projection of the primitive's position to the image plane, i.e. µ c k := Π c k (µ k ), and with covariance given by Σ c k := J c k Σ k J c⊤ k , where J c k is the Jacobian of Π c k evaluated at µ k . Finally, we apply traditional alpha compositing of the 3D Gaussians to render pixels p of camera c:EQUATIONTo optimize parameters θ of f θ , i.e. 3D Gaussian parameters µ k , α k , q k and a k , sequence latent vectors ω t s and implicit neural fields ψ and ϕ, we use an end-to-end differentiable rendering pipeline. We render both an RGB color image Î and a depth image D and apply the following loss function:EQUATIONwhere L rgb is the L1 norm, L ssim is the structural similarity index measure [79] , and L dep is the L2 norm. We use the posed training images and LiDAR measurements as the ground truth. If no depth ground-truth is available, we drop the depth-related loss from L.Pose optimization. Next to optimizing scene geometry, it is crucial to refine the pose parameters of the reconstruction for in-the-wild scenarios since provided poses often have limited accuracy [10, 17] . Thus, we optimize the residuals ∆P t s ∈ se(3), ∆P c ∈ se(3) and ∆ξ t o ∈ se(2) jointly with parameters θ. We constrain object pose residuals to se(2) to incorporate the prior that objects move on the ground plane and are oriented upright. See our supp. mat. for details on camera pose gradients.Method Residential Downtown Mean Render (s) PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ PSNR ↑ SSIM ↑ LPIPS ↓ Nerfacto +Adaptive density control. To facilitate the growth and pruning of 3D Gaussian primitives, the optimization of the parameters θ is interleaved by an ADC mechanism [18] . This mechanism is essential to achieve photo-realistic rendering. However, it was not designed for training on tens of thousands of images, and thus we develop a streamlined multi-GPU version of it. We accumulate statistics across processes and, instead of running ADC on GPU 0 and synchronizing the results, we synchronize only non-deterministic parts of ADC, i.e. the random samples drawn from the 3D Gaussians that are being split. These are usually much fewer than the total number of 3D Gaussians and thus avoids communication overhead. Next, the 3D Gaussian parameters are replaced by their updated replicas. However, this will impair the synchronization of the gradients because, in PyTorch DDP [80] , parameters are only registered once at model initialization. Therefore, we re-initialize the Reducer upon finishing the ADC mechanism in the low-level API provided in [80] .Furthermore, urban street scenes pose some unique challenges to ADC, such as a large variation in scale, e.g. extreme close-ups of nearby cars mixed with far-away buildings and sky. This can lead to blurry renderings for close-ups due to insufficient densification. We address this by using maximum 2D screen size as a splitting criterion. 1 In addition, ADC considers the world-space scale a k of a 3D Gaussian to prune large primitives which hurts background regions far from the camera. Hence, we first test if a 3D Gaussian is inside the scene bounds before pruning it according to a k . Finally, the scale of urban areas leads to memory issues when the growth of 3D Gaussian primitives is unconstrained. Therefore, we introduce a threshold that limits primitive growth while keeping pruning in place. See our supp. mat. for more details and analysis. 
experiments: 
 Datasets and metrics. We evaluate our approach across various dynamic outdoor benchmarks. First, we utilize the recently proposed NVS benchmark [17] of Argoverse 2 [81] to compare against the state-of-the-art in the multi-sequence scenario and to showcase the scalability of our method. Second, we use the established Waymo Open [23] , KITTI [21] and VKITTI2 [22] benchmarks to compare to existing approaches in single-sequence scenarios. For Waymo, we use the dynamic-32 split of [76] , while for KITTI and VKITTI2 we follow [16, 17] . We apply commonly used metrics to measure view synthesis quality: PSNR, SSIM [79] , and LPIPS (AlexNet) [82] .Implementation details. We use λ rgb := 0.8, λ ssim := 0.2 and λ depth := 0.05. We use the LiDAR point clouds as initialization for the 3D Gaussians. We first filter the points of dynamic objects using the 3D bounding box annotations and subsequently initialize the static scene with the remaining points while using the filtered points to initialize each dynamic object. We use mean voxelization with voxel size τ to remove redundant points. See our supp. mat. for more details.We compare with prior art across two experimental settings: single-sequence and multi-sequence. In the former, we are given a single input sequence and aim to synthesize hold-out viewpoints from that same sequence. In the latter, we are given multiple, heterogeneous input sequences and aim to synthesize hold-out viewpoints across all of these sequences from a single model. Multi-sequence setting. In Table 1 , we show results on the Argoverse 2 NVS benchmark proposed in [17] . We compare to state-of-the-art approaches [16, 17] and the baselines introduced in [17] . The results highlight that our approach scales well to large-scale dynamic urban scenes, outperforming previous work in performance and rendering speed by a significant margin. Specifically, we outperform [17] by more than 3 dB in PSNR while rendering more than 200× faster. To examine these results more closely, we show a qualitative comparison in Figure 3 . We see that while SUDS [16] struggles with dynamic objects and ML-NSG [17] Novel view synthesis on KITTI [21] on split from [77] . We follow the experimental protocol in [77] to compare to additional works. † baseline from [83] .Single-sequence setting. In Table 2 , we show results on the KITTI [21] and VKITTI [22] benchmarks at varying training view fractions, i.e. from dense towards sparse view setting. Furthermore, we follow the experimental protocol in [77] and show an additional comparison on the KITTI dataset with a different data split that uses also approx. 50% of views for training in Table 3 . Our approach consistently outperforms previous work as well as concurrent 3D Gaussian-based approaches [73] . In Table 4 , we show results on Waymo Open [23] , specifically on the Dynamic-32 split proposed in [76] . We outperform previous work by a large margin while our rendering speed is 700× faster than the best alternative. Note that our rendering speed increases for smaller-scale scenes. Furthermore, we show that, contrary to previous approaches, our method does not suffer from lower view quality in dynamic areas. This corroborates the strength of our contributions, showing that our method is not only scalable to large-scale, heterogeneous street data but also demonstrates superior performance in smaller-scale, homogeneous street data. We verify our design choices in both multi-and single-sequence settings. For a fair comparison, we set the global maximum of 3D Gaussians to 8 and 4.1 million, respectively. We perform these ablation studies on the residential split of [17] . We use the full overlap in the multi-sequence setting, while using a single sequence of this split for the singlesequence setting. In Table 5a , we verify the components that are not specific to the multi-sequence setting. In particular, we show that our approach to modeling scene dynamics is highly effective, evident from the large disparity in performance between the static and the dynamic variants. Next, we show that modeling appearance with a neural field is on par with the proposed solution in [18] , while being more memory efficient. In particular, when modeling view-dependent color as a per-Gaussian attribute as in [18] the model uses 8.6 GB of peak GPU memory during training, while it uses only 4.5 GB with fixed-size neural fields. Similarly, storing the parameters of the former takes 922 MB, while the latter takes only 203 MB. Note that this disparity increases with the number of 3D Gaussians per scene. Finally, we achieve the best performance when adding the generated 3D Gaussian background.We now scrutinize components specific to multi-sequence data in Table 5b . We compare the view synthesis performance of our model when i) not modeling sequence appearance or transient geometry, ii) only modeling sequence appearance, iii) modeling both sequence appearance and transient geometry. Naturally, we observe a large gap in performance between i) and ii), since the appearance changes between sequences are drastic (see Figure 3 ). However, there is still a significant gap between ii) and iii), demonstrating that modeling both sequence appearance and transient geometry is important for view synthesis from heterogeneous data sources. Finally, we provide qualitative results for non-rigid object view synthesis in Figure 4 , and show that our approach can model articulate motion without the use of domain priors. In our supp. mat., we provide further analysis.We divide our algorithm into its main components and report the individual inference runtimes across two datasets in Table 6 . While the runtime clearly correlates with scene complexity and image resolution, we observe that, on average, the runtime is dominated by scene graph evaluation and rasterization, accounting for more than 75% of the total runtime. This owes to the complexity of rasterizing millions of primitives across a high-resolution image which is computationally demanding even for efficient rasterization algorithms [18] , and handling hundreds to thousands of dynamic objects across one or multiple dynamic captures, making the retrieval of the 3D Gaussians and latent codes costly. In contrast, the queries to the neural fields account for only 13.3% of the average total runtime, making it a viable alternative to the spherical harmonics function in [18] . Overall, our method achieves interactive rendering speeds on both datasets and 20.4 FPS on average. 
conclusion: 
 We presented 4DGF, a neural scene representation for dynamic urban areas. 4DGF models highly dynamic, large-scale urban areas with 3D Gaussians as efficient geometry scaffold and compact but flexible neural fields modeling large appearance and geometry variations across captures. We use a scene graph to model dynamic object motion and flexibly compose the representation at arbitrary SUDS [16] ML-NSG [17] 4DGF (Ours) configurations and conditions. We jointly optimize the 3D Gaussians, the neural fields, and the scene graph, showing state-of-the-art view synthesis quality and interactive rendering speeds.Ground TruthLimitations and future work. While 4DGF improves novel view synthesis in dynamic urban areas, the challenging nature of the problem leaves room for further exploration. Although we model scene dynamics, appearance, and geometry variations, other factors influence image renderings in real-world captures. First, in-the-wild captures often exhibit distortions caused by the physical image formation process. Therefore, modeling phenomena like rolling shutter, white balance, motion and defocus blur, or chromatic aberrations is necessary to avoid reconstruction artifacts. Second, the assumption of a pinhole camera model in [18] persists in our work. Thus, our method falls short of modeling more complex camera models like equirectangular cameras and other sensors such as LiDAR, which may be limiting for certain capturing or simulation settings.Broader impact. We expect our work to positively impact real-world use cases like robotic simulation and mixed reality by improving the underlying technology. While we do not expect malicious uses of our method, we note that an inaccurate simulation, i.e. a failure of our system, could misrepresent the robotic system performance, possibly affecting real-world deployment. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: See above, we provide all details on data preprocessing, evaluation splits, and others.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: Not feasible with the limited computational budget.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: We provide the number and type of GPUs as well as the wall clock time to run the experiments.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes]Justification: We provide details for reproducing results, we respect all data privacy regulations, do not conduct research involving human subjects and discuss the broader impact of our work in our paper.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss this throughout the paper and dedicate a paragraph in the conclusion to it.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: We provide further details on our method and the experimental setting, as well as additional experimental results. We accompany this supplemental material with a demonstration video.We showcase the robustness of our method by rendering a complete free-form trajectory across five highly diverse sequences using the same model. Specifically, we chose the model trained on the residential split in Argoverse 2 [81] .To obtain the trajectory, we interpolate keyframes selected throughout the total geographic area of the residential split into a single, smooth trajectory that encompasses most of its spatial extent. We also apply periodical translations and rotations to this trajectory to increase the variety of synthesized viewpoints. We use a constant speed of 10 meters per second. We choose five different sequences in the data split as the references, spanning sunny daylight conditions in summer to near sunset in winter.Consequently, the appearance of the sequences changes drastically, e.g. from green, fully-leafed trees to empty branches and snow or from bright sunlight to dark clouds. Furthermore, we render each sequence with its unique set of dynamic objects, simulating various distinct traffic scenarios.We show that our model is able to perform dynamic view synthesis in all of these conditions at high quality, faithfully representing scene appearance, transient geometry, and dynamic objects in each of the conditions. We highlight that this scenario is extremely difficult, as it requires the model to generalize well beyond the training trajectories, represent totally different appearances and geometry, and model hundreds of dynamic, fast-moving objects. Despite this fact, our method produces realistic renderings, showing its potential for real-world applications.Neural field architectures. To maximize efficiency, we model ϕ and ψ with hash grids and tiny MLPs [53] . The hash grids interpolate feature vectors at the nearest voxel vertices at multiple levels.The feature vectors are obtained by indexing a feature table with a hash function. Both neural fields are given input conditioning signals ω t s ∈ R 64 and ω t o := [ω o ∈ N, γ(t) ∈ R 13 ] and output a color c among the other outputs defined in Section 3.2.For ϕ, we use the 3D Gaussian mean µ k to query the hash function at a certain 3D position yielding an intermediate feature representation f ϕ . We input the feature f ϕ , the sequence latent code ω t s , and the base opacity α k into MLP α which outputs the opacity attenuation ν s,t k . In a parallel branch, we input f ϕ , ω t s , and the viewing direction d encoded by a spherical harmonics encoding of degree 4 into the color head MLP c of ϕ that will define the final color of the 3D Gaussian.For ψ, we use a 4D hash function while using only three dimensions for interpolation of the feature vectors, effectively modeling a 4D hash grid. We use both the position µ k and the object code ω o , i.e. the object identity, as the fourth dimension of the hash grid to model an arbitrarily large number of objects with a single hash table [16] without a linear increase in memory.We input the intermediate feature f ψ and the time encoding γ(t) into the deformation head MLP χ which will output the non-rigid deformation of the object at time t, if applicable. In parallel, we input ω t s , f ψ , γ(t), and the encoded relative viewing direction d into the color head MLP c to output the final color. Note that relative viewing direction refers to the viewing direction in canonical, object-centric space. As noted in Section 3.2, the MLP heads are shared across all objects.We list a detailed overview of the architectures in Table 7 . Note that, i) we decrease the hash table size of ψ in single-sequence experiments to 2 15 as we find this to be sufficient, and ii) we use two identical networks for ψ to separate rigid from non-rigid object instances.Color prediction. The kernel function g k prevents a full saturation of the rendered color within the support of the primitive as long as the primitive's RGB color is bounded in the [0, 1] range. This can be a problem for background and other uniformly textured regions that contain large 3D Gaussians, specifically larger than a single pixel. Therefore, inspired by [84] , we use a scaled sigmoid activation function for the color head MLP c :EQUATION)where c := 0.9 is a constant scaling factor. This allows the color prediction to slightly exceed the valid [0, 1] RGB color space. After alpha compositing, we clamp the rendered RGB to the valid [0, 1] range following [18] .Time-dependent appearance. In addition to conditioning the object appearance on the sequence at hand, we model the appearance of dynamic objects as a function of time by inputting γ(t) to MLP c as described above. This way, our method adapts to changes in scene lighting that are more intricate than the general scene appearance. This could be specular reflections, dynamic indicators such as brake lights, or shadows cast onto the object as it moves through the environment.Space contraction. We use space contraction to query unbounded 3D Gaussian locations from the neural fields [54] . In particular, we use the following function for space contraction:EQUATION)For ϕ, we use ∥ • ∥ ∞ as the norm to contract the space, while for ψ we use the Frobenius norm ∥ • ∥ F . Note that we use space contraction for ψ because 3D Gaussians may extend beyond the 3D object dimensions to represent e.g. shadows, however, most of the representation capacity should be allocated to the object itself.Continuous-time object poses. Both Argoverse 2 [81] and Waymo Open [23] provide precise timing information for both the LiDAR pointclouds to which the 3D bounding boxes are synchronized, and the camera images. Thus, we treat the dynamic object poses {ξ t0 o , ..., ξ tn o } as a continuous function of time ξ o (t), i.e. we interpolate between at t a ≤ t < t b to time t to compute ξ o (t). This also allows us to render videos at arbitrary frame rates with realistic, smooth object trajectories.Anti-aliased rendering. Inspired by [59] , we compensate for the screen space dilation introduced in [18] when evaluating g c k multiplying by a compensation factor:EQUATION)where b is chosen to cover a single pixel in screen space. This helps us to render views at different sampling rates.Gradients of camera parameters. Different from [18] , we not only optimize the scene geometry but also the parameters of the camera poses. This greatly improves view quality in scenarios with imperfect camera calibration which is frequently the case in street scene datasets. In particular, we approximate the gradients w.r.t. a camera pose [R|t] as:EQUATION)This formulation was concurrently proposed in [61] , so we refer to them for a detailed derivation. We obtain the gradients w.r.t. the vehicle poses ξ via automatic differentiation [80] .Adaptive density control. We elaborate on the modifications described in Section 3.4. Specifically, we observe that the same 3D Gaussian will be rendered at varying but dominantly small scales. This biases the distribution of positional gradients towards views where the object is relatively small in Vanilla ADCFigure 5 : Qualitative comparison of ADCs. We show an example of a close-up car and observe over-smoothing when using vanilla ADC while our modified ADC leads to a sharper rendering result.view space, leading to blurry renderings for close-ups due to insufficient densification. This motivates us to use maximum 2D screen size as an additional splitting criterion.In addition to the adjustments described above and inspired by recent findings [60] , we adapt the criterion of densification during ADC. In particular, Kerbl et al. [18] use the average absolute value of positional gradient ∂L ∂µ k across multiple iterations. The positional gradient of a projected 3D Gaussian is the sum of the positional gradients across the pixels it covers:EQUATION)However, this criterion is suboptimal when a 3D Gaussian spans more than a single pixel, a scenario that is particularly relevant for large-scale urban scenes. Specifically, since the positional gradient is composed of a sum of per-pixel gradients, these can point in different directions and thus cancel each other out. Therefore, we thresholdEQUATIONas the criterion to drive densification decisions. This ensures that the overall magnitude of the gradients is considered, independent of the direction. However, this leads to an increased expected value, and therefore we increase the densification threshold to 0.0006.Hyperparameters. We describe the hyperparameters used for our method, while training details can be found in Appendix A.3. For ADC, we use an opacity threshold of 0.005 to cull transparent 3D Gaussians. To maximize view quality, we do not cull 3D Gaussians after densification stops. We use a near clip plane at a 1.0m distance, scaled by the global scene scaling factor. We set this threshold to avoid numerical instability in the projection of 3D Gaussians. Indeed, the Jacobian J c k used in g c k scales inversely with the depth of the primitive, which causes numerical instabilities as the depth of a 3D Gaussian approaches zero. For γ(t), we use 6 frequencies to encode time t.Data preprocessing. For each dataset, we obtain the initialization of the 3D Gaussians from a point cloud of the scene obtained from the provided LiDAR measurements. To avoid redundant points slowing down training, we voxelize this initial pointcloud with voxel sizes of τ := 0.1m and τ := 0.15m for the single-and multi-sequence experiments, respectively. We use the provided 3D bounding box annotations to filter points belonging to dynamic objects, to initialize the 3D Gaussians for each object, and as our object poses ξ.For KITTI and VKITTI, we follow the established benchmark used in [16, 83, 17, 73] . We use the full resolution 375 × 1242 images for training and evaluation and evaluate at varying training set fractions. For Argoverse 2, we follow the experimental setup of [17] . In particular, we use the full resolution 1550 × 2080 images for training and evaluation and use all cameras of every 10th temporal frame as the testing split. Note that we used the provided masks from [17] to mask out parts of the ego-vehicle for both training and evaluation. For Waymo Open, we follow the experimental setup of EmerNeRF [76] . We use the three front cameras (FRONT, FRONT_LEFT, FRONT_RIGHT) and resize the images to 640 × 960 for both training and evaluation. We use only the first LiDAR return as initial points for our reconstruction. We follow [76] and evaluate the cameras of every 10th temporal frame. For separate evaluation of dynamic objects, we compute masks from the 2D ground truth camera bounding boxes. We keep only objects exceeding a velocity of 1 m/s to filter for potential sensor and annotation noise. We determine the velocities from the corresponding 3D bounding box annotations. Note also that [76] do not undistort the input images, and we follow this setup for a fair comparison.Implementation details. For L dep , we use only the LiDAR measurements at the time of the camera sensor recording as ground truth to ensure dynamic objects receive valid depth supervision. We implement our method in PyTorch [80] with tools from nerfstudio [85] . For visualization of the depth, we use the inferno_r colormap and linear scaling in the 1-82.5 meters range.During training, we use the Adam optimizer [86] with β 1 := 0.9, β 2 := 0.999. We use separate learning rates for each 3D Gaussian attribute, the neural fields, and the sequence latent codes ω t s . In particular, for means µ, we use an exponential decay learning rate schedule from 1.6 • 10 -5 to 1.6 • 10 -6 , for opacity α, we use a learning rate of 5 • 10 -2 , for scales a and rotations q, we use a learning rate of 10 -3 . The neural fields are trained with an exponential decay learning rate schedule from 2.5 • 10 -3 to 2.5 • 10 -4 . The sequence latent vectors ω t s are optimized with a learning rate of 5 • 10 -4 . We optimize camera and object pose parameters with an exponential decay learning rate schedule from 10 -5 to 10 -6 . To counter pose drift, we apply weight decay with a factor 10 -2 . Note that we also optimize the height of object poses ξ. We follow previous works [87, 85, 17] and optimize the evaluation camera poses when optimizing training poses to compensate for pose errors introduced by drifting geometry through optimized training poses that may contaminate the view synthesis quality measurement.In our multi-sequence experiments in Table 1 and Table 5 , we train our model on 8 NVIDIA A100 40GB GPUs for 125,000 steps, taking approximately 2.5 days. In our single-sequence experiments, we train our model on a single RTX 4090 GPU for several hours. On Waymo Open, we train our model for 60,000 steps while for KITTI and VKITTI2 we train the model for 30,000 steps. For (b) Ablation on deformation head χ. We compare results with and without modeling non-rigid motion as deformations.Table 8 : Addtional ablation studies. In (a) we show results on a single sequence of the residential area in our benchmark and the full residential area. In (b) we use a subset of the Dynamic-32 split from [76] , i.e. the 12 sequences with the highest number of non-rigid objects. our single-sequence experiments in Table 5 we use a schedule of 100,000 steps. We chose a longer schedule for Waymo Open and Argoverse 2 since the scenes are more complex and contain about 5 -10× as many images as the sequences in KITTI and VKITTI2.We linearly scale the number warm-up steps, the steps per ADC, and the maximum step to invoke ADC with the number of training steps. For multi-GPU training, we reduce these parameters linearly with the number of GPUs. However, we observed that scaling the learning rates linearly does perform subpar to the initial learning rates in the multi-GPU setup, and therefore we keep the learning rates the same across all experiments.Additional ablation studies. In Table 8a , we show that while our approach benefits from highquality 3D bounding boxes, it is robust to noise and achieves a high view synthesis quality even with noisy predictions acquired from a 3D tracking algorithm [88] . In Table 8b , we demonstrate that the deformation head yields a small, albeit noticeable improvement in quantitative rendering results. This corroborates the utility of deformation head χ beyond the qualitative examples shown in Figures 4 and 10 . Note that the threshold to distinguish between dynamic and static areas is 1m/s following [76] so that some instances like slow-moving pedestrians will be classified as static. Also, since non-rigid entities usually cover only a small portion of the scene, expected improvements are inherently small. Table 9 : Ablation study on ADC. We compare the performance of vanilla ADC to our modified variant in the single-sequence setting.In Table 9 , we show that our modified ADC increases view quality in general, and perceptual quality in particular as it avoids blurry close-up renderings. Note that our ADC leads to roughly twice the number of 3D Gaussians belonging to objects compared to vanilla ADC, thus avoiding insufficient densification. We also show a qualitative example in Figure 5 , illustrating this effect. The close-up car rendering is significantly sharper using the modified ADC. Note that for both variants, we use the absolute gradient criterion (see Appendix A.2) for a fair comparison.Additional runtime and model analysis. In Figure 7 , we provide a comparison in terms of time per query of neural fields of different sizes (equivalent to the sizes of our ϕ and ψ) versus querying a spherical harmonics function of degree 3 as used in 3DGS [18] . We observe that the SH function is approx. 6 to 8× faster to evaluate than the neural fields. However, we show that this does not lead to a critical increase in overall runtime in Table 6 . Additionally, we note that the SH function is limited in representation capacity: It is not capable of handling varying appearance across input sequences (weather, time of day, season), transient geometry (construction sites, tree leaves), articulated motion of dynamic objects (pedestrians, cyclists), and large-scale scenes with several millions of 3D Gaussians due to memory constraints. Thus, we emphasize that the use of neural fields does not merely improve memory footprint, but enables applications that 3DGS [18] is not capable of modeling.In Figure 8 , provide an analysis on the distribution of 3D Gaussian scales across a large-scale urban scene reconstruction. We notice that while the vast majority of 3D Gaussians is small (< 0.001 mean scale), there is a small number of very large 3D Gaussians (> 1.0 mean scale) compared to the scene bounds that are approximately within [-1.0, 1.0]. This can lead to fog-like artifacts in free-viewpoint renderings. We note that this can be mitigated using a regularization term on the 3D Gaussian scales, however, we did not observe an quantitative improvement in view synthesis metrics and thus we did not include it in the experiments.Qualitative results. We provide an additional qualitative comparison of the variants iii) and ii) introduced in Section 4.2, i.e. our model with and without transient geometry modeling. In Figure 6 , we show multiple examples confirming that iii) indeed models transient geometry such as tree leaves or temporary advertisement banners (bottom left), and effectively mitigates the severe artifacts present in the RGB renderings of ii). Furthermore, the depth maps show that iii) faithfully represents the true geometry, while ii) lacks geometric variability across sequences.In addition, we show qualitative comparisons to the state-of-the-art in Figure 9 . Our method continues to produce sharper renderings than the previous best-performing method [17] , while also handling articulated objects such as pedestrians which are missing in the reconstruction of previous works (bottom two rows). Finally, we show another temporal sequence of evaluation frames in Figure 10 .Our method handles unconstrained motions and can also reconstruct more complicated scenarios such as a pedestrian carrying a stroller (right), or a grocery bag (left).SUDS [16] ML-NSG [17] 4DGF (Ours) Ground Truth Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.Note that while this criterion was described in[18], it was not used in the experiments. 


Decision: The paper proposed a new 3D scene representation that combines the strength of 3DGS and neural field to model dynamic urban scenes, which leads to better efficiency, lower memory consumption, and higher quality on benchmark datasets, while being able to handle the transient geometry and appearance. The decomposition of dynamic and static components further helps to model non-rigid motion, such as pedestrians. These are valuable contributions to our research community.  

The paper received mixed reviews initially, but the rebuttal and following discussions successfully addressed the concerns of all the reviewers who all became positive after the rebuttal. After discussing with SCA, the AC agrees with the reviewers' opinions on accepting this paper.  As promised during the rebuttal and discussion, please add the additional results of running time comparisons and analysis provided during the rebuttal to the main paper.

Reviews: review: paper_summary: paper_summary: This paper works on novel view synthesis (NVS) for large-scale, dynamic urban scenes. This paper proposes a neural scene representation called 4DGF, which uses 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. The proposed method integrates scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. The method significantly outperforms baselines in terms of speed and rendering quality on three benchmarks.
strength_weakness: strength_weakness: 1. I appreciate the authors' including a video in the submission. I found sometimes there's a large foggy region near the camera (e.g., the regions on the right during the 5-6th second), do the authors have any explanations on that? Is it caused by any limitations discussed in Sec. 5?

2. I understand that this paper mainly focuses on large dynamic scenes. I am curious how this hybrid representation performs on 3D statics scenes (e.g., the benchmarks that the original 3DGS have been tested on). This seems to be a more straightforward way to see the effect of using neural fields instead to model appearance.
questions: questions: Minor questions/suggestions:

What does GPU memory in Tab. 4 (a) mean? Is it peak memory?

In Fig. 1 inputs, the blue/orange colors for the image boundaries are also used for "geometry" and "appearance" respectively. I assume there's not such a correspondence between the input and geometry/appearance. So maybe you could change the input boundaries to different colors.

FIg. 2: extra space in "Scene configuration"

Tab. 4 appears before Tab. 3, maybe you could switch the order.
limitations: limitations: The limitations seem to have been sufficiently discussed.
paper_summary: paper_summary: The paper presents a novel 3D scene representation for novel view synthesis (nvs) in dynamic urban environments where, in particular, under heterogeneous imaging environments. The proposed representation relies on existing ingredients: 3D Gaussian Splatting, learned static/dynamic object instances, and  a global scene graph.

The resulting system yields very strong results on a series of public autonomous driving benchmarks.
strength_weakness: strength_weakness: ### - (1) Positioning of the conceptual contribution vs. the competitive landscape.

In particular, the proposed method looks very much like a revisit of Panoptic Radiance Fields [49] by replacing the NeRF component byt 3D Gaussian splats. 

While this is perfectly fine, this merits a targeted, transparent discussion in the main paper to help the reader understand the whereabouts of how the proposed contribution relates (or not) with such pieces of litterature.

### - (2) How much does it cost?

Missing piece of information regarding the resource usage, memory footprint, typical timings etc to better understand the downsides of using the provided method.

### - (3) (To a lesser extent) Certain contents in the paper are unclear.

Figure 4: what is happening? Adding color annotations or boxes would definitely help.
questions: questions: I do not have more questions or suggestions than the ones underlying the aforementioned weaknesses.
limitations: limitations: The authors provide one dedicated paragraph that reasonably addresses such considerations.
paper_summary: paper_summary: This paper proposes a hybrid neural scene representation for dynamic urban driving scene modelling. The method utilizes 3D Gaussians as an efficient geometric scaffold and neural fields to represent appearance, thereby reducing memory. To account for transient scene geometry variations caused by weather, seasons, and other factors, the authors introduce an opacity attenuation field that modulates the scene geometry. For modeling dynamic actors in the scene, an object-centric representation is used, with a non-rigid deformation in the canonical space to animate objects such as pedestrians. Experiments demonstrate that the proposed method achieves state-of-the-art performance while rendering faster than previous methods.
strength_weakness: strength_weakness: * The rendering of the proposed scene representation requires query appearance from the neural fields, it is unclear whether this will impact rendering speed compared to spherical harmonics representation.
* This paper lacks a comparison with recent neural field baselines such as UniSim and NeuRAD for urban driving scenes. Additionally, there is no comparison of the speed to 3D Gaussian baselines.
* How to control the non-rigid objects in the scene? e.g., animating the pedestrians given a sequence of human poses.
* Is it feasible to render other sensor modalities in autonomous driving, such as LiDAR?
questions: questions: This paper addresses a practical and important problem in autonomous driving. The writing is clear, and the results are promising. I look forward to the authors' response to the concerns I raised above.
limitations: limitations: The authors discussed the limitations of modeling other sensor phenomena such as rolling shutter effects, motion, and more complex camera models. They also discussed the broader societal impacts of their work.
paper_summary: paper_summary: This paper aims to perform view synthesis for dynamic urban scenes. This paper adopts 3DGS as scene geometry and uses neural fields to model the dynamic appearance of urban scenes. The neural scene graph is introduced to handle the movement of dynamic objects, and a deformation field is used to handle local articulated motions. Experiments show that the proposed approach outperforms baseline methods.
strength_weakness: strength_weakness: 1. The method presented in the paper takes 0.17 seconds to render an image at a resolution of 1550x2048, which is significantly slower than conventional 3DGS. Is the trade-off of such a significant sacrifice in rendering speed for quality improvement justified? Does the author have any solutions to address this issue?
2. The paper needs to evaluate the extent to which neural fields impact the rendering speed of 3DGS.
3. The pipeline figure of the paper should be clearer. The connections between the various modules are not easily discernible from the figure and its caption. For instance, it is not clearly depicted how the latent codes obtained from the scene configuration are inputted into the neural fields. Then, how are neural fields combined with 3DGS to represent static scenes and dynamic objects? The figure only shows simple association arrows. However, these modules are not merely input-output relationships. There are some combination operations between them.
4. The paper uses neural fields to represent appearance, which reduces the memory footprint but may also significantly impact rendering speed. Has the paper considered how to address this issue?
5. In Figure 2 of the paper, regarding the neural fields section, the symbols for static opacity correction and dynamic deformation are inconsistent with the descriptions in Section 3.2 of the paper. This is quite confusing.
6. I am curious whether the combination of neural fields with 3DGS could make the optimization of 3DGS unstable?
7. The non-rigid objects mentioned in the paper refer to cars, right? Or other objects? I did not see how the paper describes the modeling of cars. Although the paper mentions the use of scene graphs for modeling, I did not see how dynamic cars are represented using scene graphs. Does the paper treat dynamic cars as non-rigid objects directly? In this case, how can the large range of movement of dynamic cars be handled?
questions: questions: The presentation of this paper should be improved. Some important technical details are missing. The limitations from the introduction of neural fields should be discussed.
limitations: limitations: The limitations from the introduction of neural fields should be discussed.

Comments: title: Response to Rebuttal
comment: Dear Reviewers, 

We are grateful for the several suggestions and pointers to improve our paper and are happy that we could address the concerns raised. Thank you once again for reviewing our manuscript and participating in the discussion.

Best regards,
The Author(s)
We thank the reviewer for the clarification. We will integrate the line clarifying the runtime and resource usage compared to the competition including typical ranges in the camera ready version as suggested.
Dear Authors,

Having read (all of) the rebuttal contents, here is my response.

Most of my concerns have been adequately addressed but there is one misunderstanding regarding point 2 in your reply:
My point was refering to runtime and resource usage numbers wrt the competition, in particular [49] with indicative metrics, even at the top level. 

1 and 2 above address this without typical ranges, this is what is missing and I would strongly suggest to integrate one line in the camera ready version to clarify this as it is an additional plus favoring the proposed contribution.

I will maintain my positive rating.

Warm regards.
I have read through all reviewers' comments and the rebuttals by authors. My concerns have been addressed, and I would like to keep my original score.
After reading the rebuttal and other reviewers' comments, I think that this paper reaches the bar of NeurIPS.