Title: Dynamic 3D Gaussian Fields for Urban Areas
Venue: NeurIPS 2024
Accepted: yes
Keywords: Neural Rendering

Gaussian Splatting

Dynamic Urban Areas
Abstract: We present an efficient neural 3D scene representation for novel-view synthesis (NVS) in large-scale, dynamic urban areas. Existing works are not well suited for applications like mixed-reality or closed-loop simulation due to their limited visual quality and non-interactive rendering speeds. Recently, rasterization-based approaches have achieved high-quality NVS at impressive speeds. However, these methods are limited to small-scale, homogeneous data, i.e. they cannot handle severe appearance and geometry variations due to weather, season, and lighting and do not scale to larger, dynamic areas with thousands of images. We propose 4DGF, a neural scene representation that scales to large-scale dynamic urban areas, handles heterogeneous input data, and substantially improves rendering speeds. We use 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. We integrate scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. This decomposed approach enables flexible scene composition suitable for real-world applications. In experiments, we surpass the state-of-the-art by over 3 dB in PSNR and more than 200x in rendering speed.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to perform view synthesis for dynamic urban scenes. This paper adopts 3DGS as scene geometry and uses neural fields to model the dynamic appearance of urban scenes. The neural scene graph is introduced to handle the movement of dynamic objects, and a deformation field is used to handle local articulated motions. Experiments show that the proposed approach outperforms baseline methods.\n\nSTRENGTHS:\n1. The presented pipeline well handles the dynamic appearance of urban scenes.\n2. The experiments are sufficient and validate the effectiveness of the proposed approach.\n3. The idea of combining neural fields with 3DGS is sound and effective.\n\nWEAKNESSES:\n1. The method presented in the paper takes 0.17 seconds to render an image at a resolution of 1550x2048, which is significantly slower than conventional 3DGS. Is the trade-off of such a significant sacrifice in rendering speed for quality improvement justified? Does the author have any solutions to address this issue?\n2. The paper needs to evaluate the extent to which neural fields impact the rendering speed of 3DGS.\n3. The pipeline figure of the paper should be clearer. The connections between the various modules are not easily discernible from the figure and its caption. For instance, it is not clearly depicted how the latent codes obtained from the scene configuration are inputted into the neural fields. Then, how are neural fields combined with 3DGS to represent static scenes and dynamic objects? The figure only shows simple association arrows. However, these modules are not merely input-output relationships. There are some combination operations between them.\n4. The paper uses neural fields to represent appearance, which reduces the memory footprint but may also significantly impact rendering speed. Has the paper considered how to address this issue?\n5. In Figure 2 of the paper, regarding the neural fields section, the symbols for static opacity correction and dynamic deformation are inconsistent with the descriptions in Section 3.2 of the paper. This is quite confusing.\n6. I am curious whether the combination of neural fields with 3DGS could make the optimization of 3DGS unstable?\n7. The non-rigid objects mentioned in the paper refer to cars, right? Or other objects? I did not see how the paper describes the modeling of cars. Although the paper mentions the use of scene graphs for modeling, I did not see how dynamic cars are represented using scene graphs. Does the paper treat dynamic cars as non-rigid objects directly? In this case, how can the large range of movement of dynamic cars be handled?\n\nQUESTIONS:\nThe presentation of this paper should be improved. Some important technical details are missing. The limitations from the introduction of neural fields should be discussed.\n\nLIMITATIONS:\nThe limitations from the introduction of neural fields should be discussed.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a hybrid neural scene representation for dynamic urban driving scene modelling. The method utilizes 3D Gaussians as an efficient geometric scaffold and neural fields to represent appearance, thereby reducing memory. To account for transient scene geometry variations caused by weather, seasons, and other factors, the authors introduce an opacity attenuation field that modulates the scene geometry. For modeling dynamic actors in the scene, an object-centric representation is used, with a non-rigid deformation in the canonical space to animate objects such as pedestrians. Experiments demonstrate that the proposed method achieves state-of-the-art performance while rendering faster than previous methods.\n\nSTRENGTHS:\n* The paper is well-written and easy to follow.\n* The decomposed representation of appearance significantly reduces memory usage.\n* It models transient scene appearance and geometry, as well as non-rigid objects like pedestrians.\n* The evaluation and ablation study are comprehensive.\n* The paper demonstrates visually superior results compared to baselines such as SUDS and ML-NSG.\n\nWEAKNESSES:\n* The rendering of the proposed scene representation requires query appearance from the neural fields, it is unclear whether this will impact rendering speed compared to spherical harmonics representation.\n* This paper lacks a comparison with recent neural field baselines such as UniSim and NeuRAD for urban driving scenes. Additionally, there is no comparison of the speed to 3D Gaussian baselines.\n* How to control the non-rigid objects in the scene? e.g., animating the pedestrians given a sequence of human poses.\n* Is it feasible to render other sensor modalities in autonomous driving, such as LiDAR?\n\nQUESTIONS:\nThis paper addresses a practical and important problem in autonomous driving. The writing is clear, and the results are promising. I look forward to the authors' response to the concerns I raised above.\n\nLIMITATIONS:\nThe authors discussed the limitations of modeling other sensor phenomena such as rolling shutter effects, motion, and more complex camera models. They also discussed the broader societal impacts of their work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel 3D scene representation for novel view synthesis (nvs) in dynamic urban environments where, in particular, under heterogeneous imaging environments. The proposed representation relies on existing ingredients: 3D Gaussian Splatting, learned static/dynamic object instances, and  a global scene graph.\n\nThe resulting system yields very strong results on a series of public autonomous driving benchmarks.\n\nSTRENGTHS:\n### + Readability.\nOverall, in its current state, the paper's readability is relatively good. The main ideas, concepts, are mostly well discussed, conveyed, and articulated, throughout the paper.\n\n### + Practical usefullness of the considered problem.\n\n### + Structure, and Organization of the Contents.\nThe presentation is mostly on point and each dedicated section of the paper is properly balanced. The use of text real-estate is fair.\n\n### + Relative simplicity of the conceptual contribution.\n\n### + The amount of implementation details is very good.\n\n### + The reported performance.\n\n### + Implementation details for reproducibility: excellent.\n\nWEAKNESSES:\n### - (1) Positioning of the conceptual contribution vs. the competitive landscape.\n\nIn particular, the proposed method looks very much like a revisit of Panoptic Radiance Fields [49] by replacing the NeRF component byt 3D Gaussian splats. \n\nWhile this is perfectly fine, this merits a targeted, transparent discussion in the main paper to help the reader understand the whereabouts of how the proposed contribution relates (or not) with such pieces of litterature.\n\n### - (2) How much does it cost?\n\nMissing piece of information regarding the resource usage, memory footprint, typical timings etc to better understand the downsides of using the provided method.\n\n### - (3) (To a lesser extent) Certain contents in the paper are unclear.\n\nFigure 4: what is happening? Adding color annotations or boxes would definitely help.\n\nQUESTIONS:\nI do not have more questions or suggestions than the ones underlying the aforementioned weaknesses.\n\nLIMITATIONS:\nThe authors provide one dedicated paragraph that reasonably addresses such considerations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper works on novel view synthesis (NVS) for large-scale, dynamic urban scenes. This paper proposes a neural scene representation called 4DGF, which uses 3D Gaussians as an efficient geometry scaffold while relying on neural fields as a compact and flexible appearance model. The proposed method integrates scene dynamics via a scene graph at global scale while modeling articulated motions on a local level via deformations. The method significantly outperforms baselines in terms of speed and rendering quality on three benchmarks.\n\nSTRENGTHS:\n1. The idea of combining Gaussian Splatting and neural fields to model geometry and appearance, respectively, is very interesting. This makes a lot of sense considering the efficiency and the advantages of each of the two representation. This is definitely a more scalable approach to large-scale scenes compared to prior work.\n\n2. Extensive experiments have been conducted to validate the proposed method, this includes comparing with recent baselines on three benchmarks and the ablation studies that carefully examine each component. Moreover, the rendering quality improvement and the speedup is very significant on all three datasets.\n\n3. The paper is very well-written and easy to follow. Implementation details are sufficiently discussed for reproducibility.\n\nWEAKNESSES:\n1. I appreciate the authors' including a video in the submission. I found sometimes there's a large foggy region near the camera (e.g., the regions on the right during the 5-6th second), do the authors have any explanations on that? Is it caused by any limitations discussed in Sec. 5?\n\n2. I understand that this paper mainly focuses on large dynamic scenes. I am curious how this hybrid representation performs on 3D statics scenes (e.g., the benchmarks that the original 3DGS have been tested on). This seems to be a more straightforward way to see the effect of using neural fields instead to model appearance.\n\nQUESTIONS:\nMinor questions/suggestions:\n\nWhat does GPU memory in Tab. 4 (a) mean? Is it peak memory?\n\nIn Fig. 1 inputs, the blue/orange colors for the image boundaries are also used for \"geometry\" and \"appearance\" respectively. I assume there's not such a correspondence between the input and geometry/appearance. So maybe you could change the input boundaries to different colors.\n\nFIg. 2: extra space in \"Scene configuration\"\n\nTab. 4 appears before Tab. 3, maybe you could switch the order.\n\nLIMITATIONS:\nThe limitations seem to have been sufficiently discussed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}