DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos

Authors: linhan  wang
kai  cheng
shuo  lei
shengkun  wang
wei  yin
chenyang  lei
xiaoxiao  long
chang-tien  lu
virginia  tech
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 06:32:15

Abstract: We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. While neural rendering techniques have made significant strides in driving scenarios, existing methods are primarily designed for videos collected by autonomous vehicles. However, these videos are limited in both quantity and diversity compared to dash cam videos, which are more widely used across various types of vehicles and capture a broader range of scenarios. Dash cam videos often suffer from severe obstructions such as reflections and occlusions on the windshields, which significantly impede the application of neural rendering techniques. To address this challenge, we develop DC-Gaussian based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS). Our approach includes an adaptive image decomposition module to model reflections and occlusions in a unified manner. Additionally, we introduce illumination-aware obstruction modeling to manage reflections and occlusions under varying lighting conditions. Lastly, we employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Experiments on self-captured and public dash cam videos show that our method not only achieves state-of-the-art performance in novel view synthesis, but also accurately reconstructing captured scenes getting rid of obstructions.

Main paper: 
 introduction: 
 Neural Radiance Field (NeRF) [30] has revolutionized the image-based rendering area with its differentiable volume rendering technique. 3D Gaussian Splatting (3DGS) [19] pushes the frontier further with real-time rendering speed. These technologies have been applied to datasets captured by autonomous cars [42, 6, 25] , opening up numerous new possibilities in autonomous driving, such as simulating driving scenarios [57, 51] for robust training of perception models and providing effective 3D scene representations to enhance comprehensive environmental understanding [14, 63, 66] . Although these datasets provide multi-modality sensor data, their diversity in real-world driving scenarios is still limited [7] . Fortunately, dash cam videos deeply reflect the diversity and complexity of real-world traffic scenarios [8] . They are used to provide largescale, diverse driving video datasets in a crowdsourced manner [59] . Dash cam videos also offer unique value by capturing multi-agent driving behaviors [7] and evaluating the robustness of algorithms under visual hazards [60] . Moreover, the global dash cam market is rapidly expanding, driven by increasing awareness of vehicular safety [34] . Therefore, the exploration of utilizing dash camera data in neural rendering shows great potential, offering enormous amounts of data for autonomous driving applications.However, naively training 3DGS on dash cam videos often results in a significant deterioration of rendering quality and geometry. This degradation is primarily due to the common existence of obstructions (reflections and occlusions such as mobile phone holders and stains, as shown in Fig. 2 ) on windshields. In these scenarios, 3DGS models the obstructions as stationary geometries while they are dynamic in nature (moving with cars), thus unavoidably causing inaccurate geometry and blurry renderings in novel views.Although some single-image-based obstruction removal methods exist, directly applying them to this task is nontrivial. These obstructions arise from various sources, while existing removal methods impose strict assumptions on obstructions [12, 64, 56, 55, 38, 20] . For instance, assumptions like outof-focus [64] and ghost cue [38] allow previous methods to perform well in specific cases, but these assumptions don't always hold in dash cam videos. Moreover, the performance of learning-based methods degrades for out-of-distribution images [47, 27, 21] . Several NeRF methods [16, 33, 68, 69] attempt to reconstruct scenes with reflections by decomposing transmission (background scene) and reflection with independent NeRFs. This approach can benefit from the strong multi-view aggregation power of NeRF. However, previous methods are insufficient for dash cam videos because the obstructions on windshields do not align well with NeRF's design. The vanilla NeRF is intended for static scenes, whereas windshields and their reflected objects move with the cars.In this paper, we introduce DC-Gaussian, a method for modeling high-fidelity obstruction-free 3D Gaussian Splatting from dash cam videos. We introduce three key innovations: 1) Adaptive Image Decomposition. To clearly decompose images with complex reflections and occlusions, we propose an adaptive image decomposition approach. We use an opacity map to learn the transmittance of the windshield, which adaptively estimates the contribution of the background scene to the image. 2) Illumination-aware Obstruction Modeling(IOM). We observe that the obstructions are mainly caused by the objects being relatively static with the dash camera, but the obstructions present varying effects due to changing illumination. We therefore propose modeling global obstructions that are shared for all views. Moreover, a novel Latent Intensity Modulation (LIM) module is introduced to learn the illumination changes from the scene and enable synthesis of reflection with varying intensity. 3) Geometry-Guided Gaussian Enhancement(G3E). We further leverage multi-view stereo to introduce geometry prior into 3DGS training, which enhances the details and sharpness of 3DGS rendering.To evaluate the efficacy of our method, we conduct extensive experiments on public datasets [59] and a self-captured dataset. The experiments show that our method not only achieves state-of-the-art novel view synthesis, but also clearly removes obstructions from neural rendering.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: We show our method's performance on novel view synthesis and obstruction removal, reflecting our contributions. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
novel view synthesis for driving scenes: 
 Novel view synthesis aims to render novel views given posed images of the same scene. NeRF [30] , which combines multiple layer perceptions and differentiable volume rendering, initiated a revolution in this area by rendering photorealistic images. Subsequent works [3] [61] extended NeRF to unbounded large-scale scenes by warping the space into a bounded cube. BlockNeRF [44] first introduced NeRF to driving scenes by dividing the scenes into blocks and training separate models on them. The method of scene division was further improved in later works [45, 65] . To apply NeRF to multi-camera systems, UC-NeRF [10] addresses the under-calibration problem by refining the poses with spatio-temporal constraints. S-NeRF [52] uses sparse LiDAR points to enhance the training of NeRF and learn robust geometry. Decomposing dynamic objects and static backgrounds in driving scenes presents another challenge. Some works [46, 54] tackle this challenge with the help of LiDAR and 2D optical flows.Recently 3DGS [19] has attracted great attention in the research community. It achieves optimal results in novel view synthesis and rendering speed by explicitly modeling a 3D scene with 3D gaussians. Some researchers have extended it to dynamic objects and scenes. Given a set of dynamic monocular images, a deformation network is introduced to model the motion of Gaussians [58] . DrivingGaussian [67] models and decomposes dynamic driving scenes based on composite gaussian splatting. GaussianPro [9] improves the geometry of 3DGS by controlling the densification process of 3DGS with classic PatchMatching algorithm [2] . Zhou et al. [66] propose to utilize 3D Gaussian Splatting for holistic urban scene understanding. However, dash cam videos, an important data source for understanding driving scenes, remain unexplored due to obstructions on the windshield. Despite previous works [67, 54, 58] making progress in separating dynamic objects from background scenes, the image decomposition problem we are addressing presents unique challenges because of the obstructions' transparent or semi-transparent nature.Single-image reflection removal. To address the highly ill-posed problem of single-image reflection removal, various methods leverage different cues. Polarization cues are particularly valuable as they are inherently present in all natural light sources [36, 13, 21, 28] . Gradient priors [23, 24, 1] are utilized based on the observation that reflection and background layers often exhibit different levels of sharpness. Additionally, ghosting cues [38, 64, 18] and flash/non-flash pairs [20, 22] can be effective in certain scenarios. However, these assumptions do not always hold in real-world situations. With the advancement of deep learning technology, learning-based methods [17, 12, 50, 17] have been developed to model reflection properties more comprehensively. Despite their success, reflection removal from a single image remains challenging due to the inherently ill-posed nature of the problem, the absence of motion cues [27] , and the difficulties in out-of-domain generalization [47] .Multi-image layer separation. Existing methods often exploit differences in motion patterns between transmission and obstruction layers and use learned image priors [15] to decompose images into multiple components. These layer separation methods estimate dense motion fields for each layer using optical flow [43] , SIFT flow [40] , and deep learning-based flow estimation methods [41, 27] . Recently, Nam et al. [32] propose a multi-frame fusion framework based on neural image representation, achieving strong performance on various layer-separation tasks. Similarly, NSF [11] fuses RAW burst image sequences by modeling optical flow with neural spline fields. However, methods designed for burst images struggle with large pixel motions in driving scenes.NeRF with reflections. NeRFRen [16] is the pioneering work that adapts NeRF to model scenes with reflections by proposing to model transmitted and reflected components with separate NeRFs. NeuS-HSR [33] achieves high-fidelity 3D surface reconstruction by explicitly modeling the glasses Global LatentMLP MLPFor supervision with an auxiliary plane module. Zhu et al. [68] introduce recurring edge cues to achieve robust results under sparse views. However, previous methods are insufficient for dash cam videos because the obstructions on windshields do not align well with NeRF's design. The vanilla NeRF is intended for static scenes, whereas windshields and their reflected objects move with the cars. The varying illumination in the wild makes reflections modeling even more challenging. In contrast to existing methods, our proposed obstruction modeling approach leverages the static nature of obstructions in camera coordinates and captures the varying intensity of reflections.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: We discuss the limitations of this paper in conclusion section. Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Our DC-Gaussian extends the standard 3DGS to corrupted dash cam videos. We begin by reviewing the standard 3DGS pipline 3.1. Then we introduce the Adaptive Image Decomposition 3.3 to decompose the reflections and occlusions from the corrupted dash cam images. In 3.3, we propose the Illuminate-aware Obstruction Modeling module. A novel Latent Intensity Modulation is introduced to enable high quality modeling of reflection even under vary illumination. Finally, the Geometry Guided Gaussian Enhancement strategy is explained in 3.4.3DGS [19] models the 3D scene as a set of 3D Gaussians. Each 3D Gaussian G is defined as:EQUATIONwhere µ and Σ represent its mean vector and covariance matrix, respectively. For optimization purpose, the covariance matrix is further expressed as Σ = RT S T R T , where S and R are the scaling matrix and rotation matrix, respectively. To render an image, the splatting technique [70] is applied. Specifically, the color of each pixel p is calculated by blending N ordered Gaussians {G i |i = 1, ..., N } overlapping p as:EQUATIONwhere α i is obtained by evaluating a projected 2D Gaussian [70] from G i at p multiplied with a learned opacity of G i , and c i is the learnable color of G i .To synthesize images with obstructions, previous methods [16, 33] render transmission image I t and obstruction image I o separately and utilize a naive linear combination of I t and I o to render the final output I, as illustrated in Eq. 3:EQUATIONwhere ϕ 1 and ϕ 2 are manually chosen. While this approach achieves descent performance on the pure reflection corrupted images, it cannot achieve good performance on dash cam videos with complex obstructions. Among the common obstructions, mobile-phone holders are opaque, stains is semi-transparent and reflections are transparent. Faced with this complex situation, inspired by [32] , we propose to learn the opacity map ϕ from the input images. As a result, we reformulate the rendering process:EQUATIONwhere u, v ∈ [0, 1] are the continuous image coordinates and j ∈ [0, 1, ..., N -1] is the frame index. Instead of defining the opacity field in 3D world space, we define the opacity relative to the 2D image space of each view, resulting in a learnable 2D tensor in practice. This approach is more convenient for modeling obstructions because view-dependent effects are only related to the training images.In this image decomposition method, the transmission images represent the driving scenes viewed through the windshield. We can use standard 3DGS to model these scenes due to its multi-view consistent property. Thus, I t (u, v, j) can be easily calculated by Eq. 2, given the camera pose of frame j. I o (u, v, j) represents the appearance of the complex obstructions on the windshield. In the implementation, we incorporate ϕ into I o (u, v, j) in the second term of Eq. 4 for robust training. We explain the modeling of I o (u, v, j) in the next section.Decomposing the images into transmission and obstruction is challenging due to the strong ambiguity between the two components. It is an ill-posed problem without prior information. We have two observations about the obstructions on the windshield that can strongly mitigate this ambiguity.Observation 1 As shown in Fig. 4 (a), the reflections on the windshields are from objects (air conditioner vent) inside the car, which means they are relatively stationary with the car [39] . Additionally, occlusions are attached to the windshields, which are also relatively stationary with the car. Consequently, we can assume that the appearance of the obstructions is like an image shared globally by all the frames in a dash cam video.Observation 2 As cars move along the road, the trees and buildings on the sides occasionally block the incident light, affecting the intensity of the reflections. For example, under strong light, reflections are also strong, as shown in Fig. 4 (a), while the reflections "disappear" under weak light, as seen in Fig. 4 (d) . Thus, the strength of the reflections is conditioned on the car's position.These two observations require us to design a model for obstructions that takes advantage of the global-sharing property of obstructions and also grasps varying intensity reflections conditioned on the car's position.Global-shared Multi-resolution Hash Encoding. To align our design with Observation 1, we use a global-shared latent representation for obstructions' appearance. Specifically, we use continuous image coordinates u, v ∈ [0, 1] as the input. Then we use a multi-resolution hash encoder γ [31] to map these coordinates into high-dimensional learnable latent features. For example, we use an LFigure 4 : When the intensity of incident light changes, the strength of reflections also changes accordingly (a, d). Our method achieves high-fidelity reflections synthesis (c, f) and reasonable decomposition results (b, e) under varying light. The reflections in (f) are too weak to be seen by the eye, so we brighten it to reveal the details.level hash encoder where each level stores F dimensional features. Thus, each u, v pair maps to an L * F dimensional latent features γ(u, v). While our method is not restricted to this specific type of spatial encoding, we choose the multi-resolution hash encoder for two reasons. First, its hierarchical multi-resolution representation can adaptively learn the obstruction appearance in a coarse-to-fine fashion. Second, its efficient implementation matches the speed of 3DGS, without slowing down the training significantly.Latent Intensity Modulation. In order to accurately capture the intensity variations in environmental lighting conditioned on car positions, we propose a novel Latent Intensity Modulation (LIM) module. Specifically, to enable selective activation of reflections conditioned on camera positions, we design a Scaling Gate ω and an Offset Gate β, which are generated by two MLPs with the concatenation of camera positions π j and γ(u, v) as input.EQUATIONThen, the latent features are modulated element-wise through these two gates. Finally, we use a MLP to decode the modulated latent features into RGB information of obstructions.EQUATIONWith the LIM module, our method achieves effective image decomposition and synthesizes highfidelity reflections under varying light conditions, as shown in Fig. 4 .In IOM, we utilize the motion pattern prior of obstructions to reduce ambiguity in decomposing images. Despite this, some ambiguity still remains. To further enhance performance, we incorporate geometry priors. Typically, strong obstructions affect only portions of the images. As cars move, the same objects in the 3D world can appear in several image fragments which are not or less interfered with by obstructions. In these image fragments, the texture in the transmission is less blurred by obstructions, and multi-view consistency is maintained. Based on this intuition, we leverage a multi-view stereo (MVS) algorithm to identify these "image fragments" and generate geometry priors for 3D Gaussians.Specifically, we employ a deep MVS method [48] to generate depth maps for all views. A geometric consistency filtering process [37] is leveraged to generate masks M j masking the multi-view consistent areas, which are essentially the "image fragments" we want to identify. The masked depth maps are then mapped to 3D space as dense point clouds, which are used to initialize 3D Gaussians at physically accurate positions. To find more multi-view consistent "image fragments," we propose suppressing the obstructions in training images by employing our proposed method. Specifically, we remove obstructions from input images to obtain transmission Îj t . For areas blocked by occlusions(where ϕ has large value), we inpaint the content with I j t .Figure 5 : Comparisons with 3DGS on novel view synthesis. Because the obstructions violate multiview consistency, the performance of 3DGS degrades significantly, resulting in artifacts and blurry renderings (highlighted by red arrows). In contrast, our method not only faithfully synthesizes novel view renderings but also renders transmission with fine details, exhibiting an improvement of 3.05dB in terms of PSNR.EQUATIONHere I j is the j th input image. I j o and I j t are synthesized by trained IOM and 3DGS, respectively. Eq. 7 is derived from Eq. 4. We use 0.5 for the threshold τ in all the experiments.We develop our method based on 3DGS. We borrow multi-resolution hash encoding and fast MLP implementation from tiny-cuda-nn [31] to build IOM. We choose PatchMatchNet [48] as the MVS method in G3E. We follow previous driving scenes reconstruction works [46, 10] to separately model the sky areas. The final loss function is:EQUATIONL pho is the same as in standard 3DGS [19] . L sky is borrowed from UCNeRF [10] . We use a L 1 loss L opacity = (u,v) ∥ϕ(u, v)∥ 1 to regularize the opacity field, where (u, v) are the image coordinates. This opacity loss encourages the opacity map to have the minimum areas that could satisfy the optimization. This design is based on the prior knowledge that opaque objects typically occupy only small portions of the windshield. We use 0.001 for both λ 1 and λ 2 . We run all the experiments with an A100 GPU. Each scene contains approximately 300 images in our evaluation datasets. Combined training of the 3DGS and IOM for 30k iterations with Adam optimizer takes about 30 minutes. It takes 40 minutes in total to evaluate MVS and run geometry filtering.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA]Answer: [Yes] Justification: We will submit our code and data to github. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark). Justification: We describe the GPU we used in implementation details.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
datasets: 
 BDD100K. To evaluate the performance of our method and baselines, we adopt BDD100K [59] for evaluation. This dataset contains 8 scenes that are from dash cam videos captured in daily life. They contain common obstructions, such as reflections, mobile phone holders, stickers and stains. Evaluation on this dataset reflects performance on real life dash cam videos. To further evaluate the performance of our method on strong reflection conditions, we established DCVR (Dash Cam Videos with Reflection) dataset. This dataset contains 10 dash cam videos we collect. The original videos are undistorted [5] to ease the structure-from-motion algorithm.We utilize the popular tools COLMAP [37] and HLoc [35, 26] to estimate the camera parameters.In both datasets, each sequence consists of approximately 300 frames, extracted from 10-second videos at a frame rate of 30 Hz. For scenes containing car hoods, the lower parts of the images are removed during preprocessing. Seven out of eighteen scenes in two datasets involve car turns, introducing diverse illumination changes. Additional details and visual results are provided in the appendix.We choose 3DGS as our baseline [19] . We also compare our method with other state-of-the-art methods Zip-NeRF [4] , NeRF-W [29] and GaussianPro [9] . We use the unofficial implementations of Zip-NeRF3 and NeRF-W4 . To evaluate the performance of novel view synthesis, following common settings [3] , we select one of every eight images as testing images and the remaining ones for training. Since our method is also designed for image decomposition, we also compare our method with state-of-the-art obstruction removal methods, including DSRNet [17] , NIR [32] and Liu et al. [27] .For the quantitative evaluation, we conduct comparison with baselines on both BDD100K and DCVR.We apply the three widely-used metrics for evaluation, i.e., PSNR, SSIM [49] , and LPIPS [62] .The results are shown in Tab. 1. The consistent superior performance of our method shows the efficacy of the proposed modules. Though GaussianPro and Zip-NeRF achieve great performance in obstruction-free scenes with their progressive propagation strategy and anti-aliasing mechanism, without separate obstruction modeling, they cannot handle the obstructions corrupted dash cam videos. NeRF-W is designed to handle illumination and content difference between images taken at different times but still cannot handle obstructions on the windshield. We show more visual results in the appendix. DC-Gaussian achieves 120 fps at a resolution of 1920x1080 on an RTX 3090 GPU. Although ours is slightly slower than 3DGS, the speed still enables real-time rendering, which is crucial for applications such as autonomous driving simulators.Novel view synthesis. We show the novel view synthesis results in Fig. 5 . Without a proper representation for obstructions, 3DGS can hardly synthesize high-quality obstructions. Moreover, its wrong geometry also results in blurry renderings and artifacts on the road surface. In contrast, our method effectively tackles the ambiguity between obstructions and transmissions and synthesizes both components with high-fidelity. We provide depth map in the appendix.Obstruction removal. We show the obstruction removal results in Fig. 6 . The single image reflection removal method (e) only marginally suppresses reflections. Multi-image layer separation methods We conduct extensive ablation studies on DCVR to explore the impact of each module in DC-Gaussian.Quantitative results are shown in Table 2 . To assess the efficacy of global-shared hash encoding, we evaluate 3DGS with a Naive Obstruction Module (NOM), where latent features are directly decoded by an MLP to generate RGB. NOM shows significant improvement over the baseline, demonstrating that global-shared hash encoding effectively leverages the static prior of obstructions in dash cam videos. The adaptive image decomposition (AD) strategy further enhances results by effectively modeling occlusions, as shown in Fig. 7 . Additionally, LIM improves performance by capturing intensity changes in reflections. Finally, incorporating geometry priors into 3D Gaussians with G3E effectively suppresses artifacts on the road and reveals sharper details, as illustrated in Fig. 8 .  
conclusions: 
 In conclusion, we propose DC-Gaussian, which effectively addresses the challenges of extending 3D Gaussian Splatting to dash cam videos for the first time. The proposed Adaptive Image Decomposition module enables unified modeling of reflections and occlusions. To handle the reflections and occlusions under challenging lighting conditions, we introduce Illumination-aware Obstruction modeling. Additionally, we employ a Geometry-Guided Gaussian Enhancement strategy to further improve rendering quality. Experiments on BDD100K and DCVR demonstrate significant improvements in rendering quality and image decomposition, setting a new benchmark for neural rendering with dash cam videos.Limitations and future work Currently, DC-Gaussian has only been evaluated on single-sequence videos. However, considering the vast amount of dash cam footage available, extending DC-Gaussian to a multi-sequence video setting and leveraging dense view images to achieve more pleasing results would be a promising direction for future research. In addition, Our method is not specifically designed to improve performance on dynamic scenes. We provide additional experimental results in the appendix. The results demonstrate that dynamic objects do not significantly impact the performance of obstruction removal. When dynamic objects move at a slow speed, our method also presents reasonable results. We plan to incorporate techniques [53] for dynamic objects modeling into our method in future research to enable robust dynamic modeling.A Appendix / supplemental material Composed Depth mapFigure 9 : Comparisons with 3DGS and Zip-NeRF [4] on novel view synthesis show that obstructions violate multi-view consistency, leading to erroneous geometry in 3DGS and Zip-NeRF, as evident in the depth maps. This results in blurry renderings and artifacts. In contrast, our method effectively addresses the ambiguity introduced by obstructions and learns physically reasonable geometry, achieving renderings with fine details. Figure 12 : Visual results on dynamic scenes. All three scenes demonstrate that dynamic objects do not significantly impact the decomposition of obstructions. Our method achieves good performance in the scene shown in the first row, where the dynamic objects are moving slowly. However, in the second and third rows, where the dynamic objects are moving at higher speeds, our method shows suboptimal performance. NeurIPS Paper Checklist 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [No] Justification: We discuss the positive societal impacts; however, we do not foresee any potential negative impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: Yes, the creators or original owners of the assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.• If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: We provide information about the dataset we use. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Justification: We understand and respect the NeurIPS Code of Ethics.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).https://github.com/SuLvXiangXin/zipnerf-pytorchhttps://github.com/kwea123/nerf_pl 


Decision: As the Area Chair for submission 4323, I have thoroughly evaluated the reviewers' feedback and the authors' responses. The consensus leads me to recommend acceptance of this paper for NeurIPS 2024.

The paper addresses novel view synthesis with dash cam videos, a topic of growing importance but limited exploration. It introduces DC-Gaussian, an innovative method that effectively tackles unique challenges such as obstructions and variable lighting in dash cam footage. The experimental results are robust, showing clear improvements over existing methods like NeRF-W, especially in handling common dash cam obstructions. 

Reviewers initially expressed concerns about the practicality and the handling of dynamic scenes. However, the authors provided comprehensive rebuttals that demonstrated the method's utility and robustness, supplemented by recent literature and additional experimental validations.

he authors are encouraged to incorporate final feedback regarding dataset details and benchmarking in their final manuscript. The acceptance of this paper will foster further research in a promising direction.

Reviews: review: paper_summary: paper_summary: This paper focuses on using dash cam videos for 3D Gaussian Splatting-based outdoor scene reconstruction. To address challenges such as reflections and occlusions on windshields, DC-Gaussian introduces an adaptive image decomposition module to model these effects in a unified manner. Additionally, an illumination-aware obstruction modeling technique is designed to manage reflections and occlusions under varying lighting conditions. Finally, the authors employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Extensive experiments on self-captured and public dash cam videos verify the effectiveness of the proposed method.
strength_weakness: strength_weakness: (1) Limited Practicality: My main concern is the practical value of this work. Dash cam videos generally provide single-view sequences, and these sparse views would result in the learned 3DGS suffering from limited novel-view synthesis quality. Additionally, the inherent limitations of dash cam videos, such as reflections and occlusions on the windshields, would further degrade rendering quality despite the authors’ efforts to mitigate these issues with developed modules. Therefore, I find it hard to believe that people would choose dash cam videos for high-quality 3D scene reconstruction in real-world applications.

(2) The authors introduce global-shared hash encoding and MLPs for illumination-aware obstruction modeling. The reviewer is curious about the running speed compared to the baseline method, 3DGS. Please provide a comparison of the inference times.
questions: questions: See weaknesses.
limitations: limitations: The authors have adequately addressed the limitations and the potential negative societal impact of their work.
paper_summary: paper_summary: This paper deals with the problem of novel view synthesis in outdoor scenes captured with a dashcam. The authors develop a 3D Gaussian Splatting based method that is robust to common obstructions observed in dashcam videos, mainly due to the way these videos are captured: Mobile-phone holders, reflections and windshield stains create artifacts in the reconstruction as they are not natively modeled by 3DGS and cannot be reconstructed as part of the static 3D scene as they move with the ego-vehicle. The authors propose a learnable opacity map that decomposes the images into obstructions and real scene and a lighting modulation module that adapts obstructions over for each view based on varying illumination across views. These are implemented with a 2D hashgrid and MLP heads. They can then use the opacity map to blend the obstruction and real scene for rendering each view. The authors also use an MVS method with geometric consistency filtering to initialize the 3D Gaussian positions.
strength_weakness: strength_weakness: - There is no description of $\mathcal{L}_{opacity}$, but without this it is not possible to judge whether the learned opacity map is learned in a self supervised manner or if additional labels are needed.
- The task setup is similar in spirit to NeRF-in-the-wild. Therefore, to judge the importance of the proposed design opposed to generic in-the-wild methods (NeRF-W,  not tailored towards dashcam videos, it would be important to understand how such methods compare on the benchmarks the authors use. However, the only baselines the authors compare to are ZipNeRF, 3DGS and GaussianPro, all of which are not designed for this type of captures.
- As far as I understand, G3E consists of running PatchMatchNet with a standard geometry consistency filter. There seems to be no innovation in that to me. What is Fig. 8 comparing exactly? Random initialization vs MVS or SfM vs MVS initialization?
questions: questions: It would be great if the authors can clarify my concerns w.r.t. the loss details, G3E and may provide more insights on the choice of baselines.
limitations: limitations: The limitation discussion is short and does not point to meaningful failure cases.
paper_summary: paper_summary: This paper presents a new Gaussian Splatting novel view synthesis method specifically for in-vehicle dash cam videos containing various obstructions. To the challenges caused by obstructions, the proposed method separately represents the transmission and obstruction part of the camera capture. The transmission part is a 3DGS representation, and the obstruction part is modeled as a 2D representation with an opacity map and position-dependent illumination-aware neural model. The synthesized transmission images are also used to enhance the geometry prior for 3DGS training. The evaluation results show apparent improvements over previous methods.
strength_weakness: strength_weakness: 1. The proposed method is more like an improvement for a specific Gaussian Splatting application. The similar components in the proposed method can be found, in part, from other related work. Therefore, the novelty of this paper is moderate.

2. In Sec. 3.2, the authors reformulate the composition equation to Eq. 4. This equation is a bit incorrect from the perspective of light transport. I feel the $(1 - \phi) I_t + (\phi) I_o $ would make more sense.

3. In the experiments, it is still unclear about the evaluation datasets. The following key informations are missing in the paper:

   * The number of scenes used and the number of frames per scene.
   * Image quality, and resolutions (since dash cam might have worse image quality).
   * Do the scenes contain dynamic objects?
   * Is there any scene containing opaque occlusions such as a car hood?
   * Do the drive sequences contain any turning to show more diverse illumination changes?      

   I would suggest the authors show more visual results in the revised version.

I am looking forward to the authors' responses.
questions: questions: * I wonder how well the proposed method works for the reflective car hoods, which are also commonly seen from the dash cam.
limitations: limitations: Limitations are discussed at the end of the paper, but I don’t think it is adequate, see my comments in the above sections.
paper_summary: paper_summary: This paper tries to reconstruct 3DGS (3D Gaussian Splatting) with Dash Cam Videos, which introduces obstructions on windshields. An adaptive image decomposition is applied to learn transmission images and obstruction images separately. The former is modeled with 3DGS in G3 Enhancement and are rendered into 2D space, while the latter is learned on 2D camera space conditioned by car positions in Illumination-aware Obstruction Modeling. Further, an opacity map is optimized to weigh these two images. This method surpasses Zip-NeRF, 3DGS and other runner-ups in public and self-collected datasets.
strength_weakness: strength_weakness: 1. The opacity map is modeled to be independent of car position, but in fact, according to Fresnel's law, opacity is also related to the light distribution, and therefore also related to the position.
2. The two observations in 3.3 are too strong. This somehow weakens the generalization of the algorithm.
questions: questions: 1. How to calculate opacity loss in Eq.8?
2. Are there any visualizations related to reconstructed 3DGS?
3. How fast is the rendering?
limitations: limitations: As stated in this paper, DC-Gaussian has only been evaluated on single-sequence videos. I would also prefer the opacity map to be detailed modeled. However, considering the good performance, this trade-off on opacity is acceptable.

Comments: title: Awaiting Your Response as Deadline Nears
Follow-Up on Rebuttal Discussion
Thanks for the Response
Thanks for the Response
Thanks for the Response
Thanks for the Response
comment: As the deadline approaches, I am eagerly anticipating your response.
We value your feedback and are eager to address any further questions or concerns you may have. If you have had a chance to review our response and have additional thoughts, we would greatly appreciate your input.
Thank you for your response. I'm glad that our explanation helped clarify your confusion. I also greatly appreciate your suggestion regarding the dataset introduction. We will certainly incorporate this feedback in the final draft.
Thank you to the authors for the response and clarification regarding Fig. 1. I realize I had misunderstood Fig. 1. After reading the authors' response and other reviewers' comments, I have started to appreciate the potential of dash cam videos in autonomous driving scenarios, although I think there are still many practical challenges, such as the limited NVS capabilities in sparse viewpoints (e.g., large rotational angle changes) and the issue of mismatched image properties (e.g., resolution) among different models of dashcams. Furthermore, as a new task, the original manuscript lacks a detailed introduction to the benchmark datasets (e.g., which sequences were selected an and the criteria for selection), which is crucial for performance evaluation. Although there have been some additions in the rebuttal, it still detracts from the paper's credibility. I suggest that the authors address this in the final draft. Overall, considering the feedback from other reviewers and the authors' responses, I am willing to raise my score in my final decision.
Thanks very much for your further response.

**“The viewpoints of Fig.1 (b) and (c) are the same, which are the not the results of novel view synthesis (NVS)”**

We respectfully disagree with this statement. The viewpoints of (b) and (c) must be identical because (b) represents the real-captured image used as a reference, while (c) is the synthesized image produced by our method after obstruction removal. In the context of novel view synthesis, it is standard practice to present synthesized images from a testing viewpoint alongside the corresponding real-captured reference images. Consistent with prior works in this field, we divided the captured images into two sets: the 'train split,' which is used to optimize Gaussian splatting, and the 'test split,' which is excluded from the optimization process but utilized for evaluating the synthesized images. For additional novel view synthesis results, please refer to our supplementary materials, where we also provide videos

**“monocular video is inherently unsuitable for NVS”**

We acknowledge that novel view synthesis (NVS) from monocular video is a challenging task, but we believe this challenge should not deter our community from making efforts to address it. In fact, NVS and 3D reconstruction from monocular cameras are long-standing problems in computer vision and computer graphics. Even in recent works related to NeRF and 3DGS, numerous studies have focused on using monocular cameras as input, such as [1,2,3,4,5,6,7,8,9,10,11]. Among these, [1, 2] are specifically designed for autonomous driving (AD) scenarios.

In addition, with the rapid advancements in the computer vision community, more challenging tasks, such as generating an entire driving sequence from a single image, have become possible through recent works [12, 13]. Notably, GenAD [13], trained on monocular videos from YouTube, leverages a substantial amount of dash cam footage. This not only underscores the significant value of dash cam videos but also motivates us to explore novel view synthesis (NVS) using less constrained data sources, such as monocular dash cam videos.


[1] Zhou, Hongyu, et al. "Hugs: Holistic urban 3d scene understanding via gaussian splatting." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[2] Lu, Fan, et al. "Urban radiance field representation with deformable neural mesh primitives." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[3] Gao, Chen, et al. "Dynamic view synthesis from dynamic monocular video." ICCV, 2021.

[4] Wu, Tianhao, et al. "D^ 2NeRF: Self-Supervised Decoupling of Dynamic and Static Objects from a Monocular Video." Advances in neural information processing systems 35 (2022): 32653-32666.

[5] Tian, Fengrui, Shaoyi Du, and Yueqi Duan. "Mononerf: Learning a generalizable dynamic radiance field from monocular videos." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[6] Das, Devikalyan, et al. "Neural parametric gaussians for monocular non-rigid object reconstruction." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[7] You, Meng, and Junhui Hou. "Decoupling dynamic monocular videos for dynamic view synthesis." IEEE Transactions on Visualization and Computer Graphics (2024).

[8] Tretschk, Edgar, et al. "Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[9] Fu, Yang, Ishan Misra, and Xiaolong Wang. "MonoNeRF: learning generalizable NeRFs from monocular videos without camera poses." International Conference on Machine Learning. PMLR, 2023.

[10] Park, Byeongjun, and Changick Kim. "Point-dynrf: Point-based dynamic radiance fields from a monocular video." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2024.

[11] Wang, Fusang, et al. "Planerf: Svd unsupervised 3d plane regularization for nerf large-scale scene reconstruction." 3DV (2023).

[12] Blattmann, Andreas, et al. "Align your latents: High-resolution video synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

[13] Yang, Jiazhi, et al. "Generalized predictive model for autonomous driving." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
Thank you for your feedback. However, this rebuttal still does not alleviate my concerns about its practicality, especially for the novel-view synthesis (NVS): 

I acknowledge the potential application value of dashcams in many visual tasks (as the authors have listed), but this does not mean it remains significant in 3D autonomous driving scene reconstruction tasks. 

Dashcams capture monocular video, and in autonomous driving scenarios, the viewpoints are typically very sparse, inherently unsuitable for NVS. Although the authors have mitigated obstruction interference, synthesizing novel views is also crucial for 3D scene reconstruction. The authors show so-called NVS in Fig. 1 (c); however, this is somewhat misleading. From my observation, the viewpoints in Fig. 1 (b) and (c) are the same, with only obstruction removal being performed, which does not qualify as NVS. Based on my experiments, novel view synthesis from monocular autonomous driving video is very challenging. Could you present your actual NVS results? I suspect they might not be very strong. 

Finally, since autonomous driving scene reconstruction is inherently a highly challenging task, I still find it hard to believe that anyone would choose dash cam videos for high-quality 3D scene reconstruction in real-world applications.
We are very happy to hear that you are satisfied with our responses.
We would like to kindly clarify that our method is specifically designed for autonomous driving scenarios not for general scenes. Dash cameras are widely used in vehicles and  dash cam footage offers unique value for autonomous driving.
Compared to general scenes, dash camera in vehicles contain some important properties that can be leveraged to tackle reflections and obstructions. As shown in main paper, the obstruction removal works designed for general scenes show quite worse results, since this task is quite ill-posed and achieving good results in specific scenes is also not that easy.  Thanks for your suggestions, we would further study this task for the goal of achieving good results in broader scenes.
Thanks for your answers to my questions. I am glad to know more details about the evaluated data. These evaluated data are sufficiently diverse to demonstrate the improvements brought by the proposed method. 

I am happy with the authors' response and will remain or possibly raise my score later.

One additional concern I had is similar to the Reviewer si9q's W1, the proposed method is practically limited to the car-mounted dash cam, but it doesn't address the obstruction challenge from the general scenes. This is still a key limiting point.
Thank you so much for your valuable feedback. We greatly appreciate your acknowledgment of this setting and your insightful suggestions for improving the modeling of the opacity map. We will conduct more experiments and analysis on the opacity map in our future work.
After thoroughly reading the PDF and the rebuttal, I would like to thank the author for adequately addressing my concerns. I will maintain my opinion that this is a very interesting setting. Since exposed cameras are highly susceptible to physical damage, I believe Dash Cam Videos present a potential solution. At least they could provide valuable prior knowledge fusion in automatic driving. 
As for the inaccuracies in opacity map modeling, there is no significant degradation in the results, so I'm willing to accept this trade-off. However, I hope the author seriously considers this issue in future work. 
I'll keep my score.