DC-Gaussian: Improving 3D Gaussian Splatting for Reflective Dash Cam Videos
Authors: Linhan Wang
Kai Cheng
Shuo Lei
Shengkun Wang
Wei Yin
Chenyang Lei
Xiaoxiao Long
Chang-Tien Lu
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural rendering; Novel view synthesis; Reflection removal
Primary area: machine_vision

Abstract: We present DC-Gaussian, a new method for generating novel views from in-vehicle dash cam videos. While neural rendering techniques have made significant strides in driving scenarios, existing methods are primarily designed for videos collected by autonomous vehicles. However, these videos are limited in both quantity and diversity compared to dash cam videos, which are more widely used across various types of vehicles and capture a broader range of scenarios. Dash cam videos often suffer from severe obstructions such as reflections and occlusions on the windshields, which significantly impede the application of neural rendering techniques. To address this challenge, we develop DC-Gaussian based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS). Our approach includes an adaptive image decomposition module to model reflections and occlusions in a unified manner. Additionally, we introduce illumination-aware obstruction modeling to manage reflections and occlusions under varying lighting conditions. Lastly, we employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Experiments on self-captured and public dash cam videos show that our method not only achieves state-of-the-art performance in novel view synthesis, but also accurately reconstructing captured scenes getting rid of obstructions.

DC-Gaussian:
Improving 3D Gaussian Splatting for Reflective Dash Cam Videos
Linhan Wang!*
Kai Cheng
Shuo Lei!
Shengkun Wang'
Wei Yin
Chenyang Lei*
Xiaoxiao Long2t
Chang-Tien Lu
Virginia Tech Hong Kong University 3UTT "CAIR, HKISI-CAS University of Adelaide
(8)
Figure Given sequence of video captured by a dash cam that may contain obstructions like reflections and occlusions, DC-Gaussian achieves high-fidelity novel view synthesis getting rid of the obstructions. (a) dash cam; (b) original video frame; (c) novel view rendering with obstruction removal.
Abstract
We present DC-Gaussian, a new method for generating novel views from vehicle dash cam videos. While neural rendering techniques have made significant strides in driving scenarios, existing methods are primarily designed for videos collected by autonomous vehicles. However, these videos limited in both quantity and diversity compared to dash cam videos, which are more widely used across various types of vehicles and capture broader range of scenarios. Dash cam videos often suffer from severe obstructions such as reflections and occlusions on the windshields, which significantly impede the application of neural rendering techniques. To address this challenge, we develop DC-Gaussian based on the recent real-time neural rendering technique 3D Gaussian Splatting (3DGS). Our approach includes an adaptive image decomposition module to model reflections and occlusions in unified manner. Additionally, we introduce illumination- aware obstruction modeling to manage reflections and occlusions under varying lighting conditions. Lastly, we employ a geometry-guidee Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Experiments on self-captured and public dash cam videos show that our method
"quual contribution Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

not only achieves state-of-the-ar performance in novel view synthesis, but also accurately reconstructing captured scenes getting rid of obstructions. See the project page for code, data:
1
Introduction
Neural Radiance Field (NeRF) 330 has revolutionized the image-based rendering area with its differentiable volume rendering technique. 3D Gaussian Splatting (3DGS) [T9] pushes the frontier further with real-time rendering speed. These technologies have been applied to datasets captured by autonomous cars 42 25). opening up numerous new possibilities in autonomous driving, such as simulating driving scenarios [1 for robust training of perception models and providing effective 3D) scene representations to enhance comprehensive environmental understanding [14
63 66
Although these datasets provide multi-modality sensor data, their diversity in real-world driving scenarios is still limited (
Fortunately, dash cam videos deeply reflect the diversity and complexity of real-world traffic scenarios 8 They are used to provide large- scale, diverse driving video datasets in crowd- sourced manner [59]. Dash cam videos also offer unique value by capturing multi-agent driv- ing behaviors [7] and evaluating the robustness of algorithms under visual hazards [60]. More-
Figure 2: Common obstructions on windshields:
over, the global dash cam market is rapidly ex- panding, driven by increasing awareness of ve-
(a) Mobile-phone holder; (b) Reflections; (c) Stains.
hicular safety [34]. Therefore, the exploration
of utilizing dash camera data neural rendering shows great potential, offering enormous amounts
of data for autonomous driving applications.
However, naively training 3DGS on dash cam videos often results in a significant deterioration of rendering quality and geometry. This degradation is primarily due to the common existence of obstructions (reflections and occlusions such mobile phone holders and stains, shown in Fig. 2 on windshields. In these scenarios, 3DGS models the obstructions as stationary geometries while they are dynamic in nature (moving with cars), thus unavoidably causing inaccurate geometry and blurry renderings in novel views.
Although some single-image-basee based obstruction removal methods exist, directly applying them this task is nontrivial. These obstructions arise from various sources, while existing removal methods impose strict assumptions on obstructions [12 64 56 55 20). For instance, assumptions like out- of-focus [64] and ghost cue (38) allow previous methods to perform well in specific cases, but these assumptions don't always hold in dash cam videos. Moreover, the performance of learning- bbsed
methods degrades for out-of- -ifrirtrinnt images [47 27 (1]] Several NeRF methods [66] 33 68 69]
attempt reconstruct scenes with reflections by decomposing transmission (background scene) and reflection with independent NeRFs. This approach can benefit from the strong multi-view aggregation power of NeRF. However, previous methods are insufficient for dash cam videos because the obstructions on windshields do not align well with NeRF's design. The vanilla NeRF is intended for static scenes, whereas windshields and their reflected objects move with the cars.
In this paper, we introduce DC-Gaussian, a method for modeling high-fidelity obstruction-free 3D) Gaussian Splatting from dash cam videos. We introduce three key innovations: Adaptive Image Decomposition. To clearly decompose images with complex reflections and occlusions, we propose an adaptive image decomposition approach. We use an opacity map to learn the transmittance of the windshield, which adaptively estimates the contribution of the background scene to the image. 2) Illmination-aware Obstruction Modeling(IOM), We observe that the obstructions are mainly caused by the objects being relatively static with the dash camera, but the obstructions present varying effects due to changing illumination. We therefore propose modeling global obstructions that are shared for all views. Moreover, a novel Latent Intensity Modulation (LIM) module is introduced to learn the illumination changes from the scene and enable synthesis of reflection with varying intensity. 3) Geometry-Guided Gaussian Enhancement (GEE). We further leverage multi-view

---

stereo to introduce geometry prior into 3DGS training, which enhances the details and sharpness of 3DGS rendering.
To evaluate the efficacy of our method, we conduct extensive experiments on public datasets [59] and self-captured dataset. The experiments show that our method not only achieves state-of-the-art novel view synthesis, but also clearly removes obstructions from neural rendering.
2 Related Work
2.1 Novel View Synthesis for Driving Scenes
Novel view synthesis aims to render novel views given posed images of the same scene. NeRF 300. which combines multiple layer perceptions and differentiable volume rendering, initiated revolution in this area by rendering photorcalistic images. Subsequent works [3] [6]] extended NeRF to unbounded large-scale scenes by warping the space into a bounded cube. BlockNeRF [44] first introduced NeRF to driving scenes by dividing the scenes into blocks and training separate models on To apply NeRF to multi-camera systems, UC-NeRF [10] addresses the under-calibration problem by refining the poses with popatio-temorre constraints. S-NeRF [52] uses sparse LiDAR points to enhance the training of NeRF and learn robust geometry. Decomposing dynamic objects and static backgrounds in driving scenes presents another challenge. Some works [46] 54] tackle this challenge with the help of LiDAR and 2D optical flows.
them. The method of scene division further improved in later works 45
65
Recently 3DGS [19] has attracted great attention in the research community. It achieves optimal results in novel view synthesis and rendering speed by explicitly modeling a 3D scene with 3D gaussians. Some researchers have extended it to dynamic objects and scenes. Given set of dynamic monocular images, a deformation network is introduced to model the motion of Gaussians (58). DrivingGaussian (67) models and decomposes dynamic driving scenes based on composite gaussian splatting. GaussianPro 9 improves the geometry of 3DGS by controlling the densification process of 3DGS with classic PatchMatching algorithm (2). Zhou al. [66) propose to utilize 3D Gaussian Splatting for holistic urban scene understanding. However, dash cam videos, an important data source for understanding driving scenes, remain unexplored due to obstructions the windshield. Despite previous works [67 [54] 58) making progress in separating dynamic objects from background scenes, the image decomposition problem we are addressing presents unique challenges because of the obstructions' transparent or semi-transpann tranprrrnnpprentt nature.
2.2 Obstruction Removal and Layer Separation
Single-image reflection removal. To address the highly ill-posed problem of single-image reflection removal, various methods leverage different cues. Polarization cues are particularly valuable as they
inherently present in all natural light sources [36] [3] 21 28). Gradient priors 23 24
utilized based on the observation that reflection and background layers often exhibit different levels of sharpness. Additionally, ghosting cues [38 64 [8] and flash/non-flash pairs (20 22 can be effective in certain scenarios. However, these assumptions do not always hold in real-world situations. With the advancement of deep learning technology, learning- based methods [17
12 50 17 have been developed to model reflection properties more comprehensively. Despite their success, reflection removal from single image remains challenging due the inherently ill- posep nature of the problem, the absence of motion cues (27), and the difficulties in out-of-domain generalization 47
Multi-image layer separation. Existing methods often exploit differences in motion patterns between transmission and obstruction layers and use learned image priors [15] to decompose images into multiple components. These layer separation methods estimate dense motion fields for each layer using optical flow 43), SIFT flow [40], and deep learning-bass flow estimation methods 41 27. Recently, Nam et al. 32 propose multi-frame fusion framework based on neural image representation, achieving strong performance on various suyer-separatie tasks. Similarly, NSF [II] fuses RAW burst image sequences by modeling optical flow with neural spline fields. However, methods designed for burst images struggle with large pixel motions in driving scenes.
NeRF with reflections. NeRFRen [6] is the pioneering work that adapts NeRF to model scenes with reflections by proposing to model transmitted and reflected components with separate NeRFs. NeuS-HSR (33) achieves high-fidelity 3D surface reconstruction by explicitly modeling the glasses
3

---

pphi
Enhancement
Geomatry Filtering
Tranemission
Update
Image
SIM
Gaussian
R Rasterizer
3DGS Initielizatiin
Opacity Map
Input images
Illumination-wware Obstruction Modeling
Composed Image
MiP JUL
Obstruction Image
Camera Coords.
Global-shares Hash E Encoding
Global
MLP
Latent Intensity Modulation
Output Images
Camera Pusition
For supervision
Figure Overview of OC-Gaussian framework. To model obstructions with different opacities in unified manner, we use an learnable opacity map to adaptively reweight the contribution of transmission. The global-shared multiresolution hash encoding is introduced fully utilize the static motion prior of obstructions. We propose Latent Intensity Modulation module grasp the intensity changes of reflections conditioned on camera positions. Finally, in the G3 Enhancement module, we geometry filtering obstruction suppptison--upppssseeede images to enhance the geometry of 3D Gaussians.
with an auxiliary plane module. Zhu et al. [68 introduce recurring edge cues to achieve robust results under sparse views. However, previous methods are insufficient for dash cam videos because the obstructions on windshields do not align well with NeRF's design. The vanilla NeRF is intended for static scenes, whereas windshields and their reflected objects move with the cars. The varying illumination in the wild makes reflections modeling even more challenging. In contrast to existing methods, our proposed obstruction modeling approach leverages the static nature of obstructions in camera coordinates and captures the varying intensity of reflections.
Method
Our DC-Gaussian extends the standard 3DGS to corrupted dash cam videos. We begin by reviewing the standard 3DGS pipline [.1]] Then we introduce the Adaptive Image Decomposition [.3 to decompose the reflections and occlusions from the corrupted dash cam images. In[3.3 we propose the Illuminate-awan Obstruction Modeling module. A novel Latent Intensity Modulation is introduced to enable high quality modeling of reflection even under vary illumination. Finally, the Geometry Guided Gaussian Enhancement strategy explained in 3.4
3.1 Preliminary 3D Gaussian Splatting
3DGS 19 models the 3D scene as a set 3D Gaussians. Each 3D Gaussian G is defined as:
G(x) e (==p)) "x (a=p)
(1)
where y and Σ represent its mean vector and covariance matrix, respectively. For optimization purpose, the covariance matrix further expressed as RT R" where and R are scaling matrix and rotation matrix, respectively. To render an image, the splatting technique (70 applied. Specifically, the color of each pixel p calculated by blending N ordered Gaussians {G = N} overlapping as:

---

c(p)
Σ c i=1
II
[1 a²)
(2)
where a; is obtained by evaluating projected 2D Gaussian 70 from G; p multiplied with learned opacity of Gū and c; is the learnable color of G..
3.2 Adaptive Image Decomposition
To synthesize images with obstructions, previous methods 33 render transmission image Le and obstruction image Io separately and utilize a naive linear combination of L, and Lo to render the final output I, illustrated in Eq.
I= x + 02 I..
(3)
where Φi and Φ are manually chosen. While this approach achieves descent performance on pure reflection corrupted images, it cannot achieve good performance dash cam videos with complex obstructions. Among the common obstructions, mobile-phone holders are opaque, stains is semi-transparent and reflections are transparent. Faced with this complex situation, inspired by [32, we propose to learn the opacity map & from the input images. As result, we reformulate the rendering process:
I(u, B, = (11 v, + x= Io(u,,,))
(4)
where u, v E [0, 1] are the continuous image coordinates and E 00 1,.,, N 1] the frame index. Instead of defining the opacity field in 3D world space, we define the opacity relative to the 2D image space of each view, resulting a learnable 2D tensor in practice. This approach is more convenient for modeling obstructions because view-dependent effects are only related to the training images. In this image decomposition method, the transmission images represent the driving scenes viewed through the windshield. We can standard 3DGS to model these scenes due to its multi-vier view consistent property. Thus, [u,u, v, can be easily calculated by Eq. 2 given the camera pose of frame j. [u,u, v, represents the appearance of the complex obstructions on the windshield. In the implementation, we incorporate ø into (u, v,j) in the second term of Eq. for robust training. We explain the modeling of (u, v,j) in the next section.
3.3 Illumination-awar Obstruction Modeling
Decomposing the images into transmission and obstruction challenging due the strong ambiguity between the two components. It is an ill-posed problem without prior information. We have two observations about the obstructions on the windshicld that can strongly mitigate this ambiguity.
Observation As shown in Fig. (a), the reflections on the windshields are from objects (air conditioner vent) inside the car, which means they are relatively stationary with the car [39] Addi- tionally, occlusions are attached to the windshields, which are also relatively stationary with the car. Consequently, we can assume that the appearance of the obstructions is like an image shared globally by all the frames in dash cam video.
Observation 2 As the incident light, affecting the intensity of the reflections. For example, under strong light, reflections are also strong, as shown Fig. (a), while the reflections "disappear" under weak light, as seen in Fig. 4 (d). Thus, the strength of the reflections is conditioned on the car's position.
along the road, the and buildings on the sides occasionally block
These two observations require to design model for obstructions that takes advantage of the global-sharing property of obstructions and also grasps varying intensity reflections conditioned on the car's position.
Global-shared Multi-resolution Hash Encoding, To align our design with Observation we use global-shared latent representation for obstructions' appearance. Specifically, we use continuous image coordinates u, v E [o, 1] as the input. Then we use a multi-resolution hash encoder 7 [1 to map these coordinates into high sigh-dimensional Icarnable latent features. For example, we use L
5

---

Strong light
Weak light
Reference
Transmission
Obstruction
Figure 4: When the intensity of incident light changes, the strength of reflections also changes accordingly (a, d). Our method achieves high-fidelity reflections synthesis (c, f) and reasonable decomposition results (b, under varying light. The reflections in (f) are too weak to be seen by the eye, so we brighten it to reveal the details.
level hash encoder where each level stores F dimensional features. Thus, each u, v pair maps to an L * I' dimensional latent features (u, v). While our method is not restricted to this specific type of spatial encoding, we choose the multi-resolution hash encoder for two reasons. First, its hierarchical multi-resolution representation
adaptively learn the obstruction appearance in coarse-to coarse-to-finn -ine fashion. Second, its efficient implementation matches the speed of 3DGS, without slowing down the training significantly.
Latent Intensity Modulation. order to accurately capture the intensity variations in environmental lighting conditioned on car positions, we propose novel Latent Intensity Modulation (LIM) module. Specifically, to enable selective activation of reflections conditioned on camera positions, we design Scaling Gate and an Offiset Gate B, which are generated by two MLPs with the concatenation of camera positions ; and 7(, v) as input.
(a, u ,T MLP(( j. 7(u, w)]), B(u, u,)) MLP(( (uu +)]),
(5)
Then, the latent features are modulated element-wing wise through these two gates. Finally, we use a MLP to decode the modulated latent features into RGB information of obstructions.
(u, v, TT2) MLP u, RTy) O (u,, e) =8(,, u, U,,,)
(6)
With the LIM module, our method achieves effective image decomposition and synthesizes high- fidelity reflections under varying light conditions, as shown in Fig.
3.4 Guodetry-Guided Gaussian Enhancement
In IOM, utilize the motion pattern prior of obstructions reduce ambiguity in decomposing images. Despite this, some ambiguity still remains. To further enhance performance, we incorporate geometry priors. Typically, strong obstructions affect only portions of the images. As move, the same objects in the 3D world can appear in several image fragments which are not or less interfered with by obstructions. In these image fragments, the texture in the transmission is less blurred by obstructions, and multi-view consistency is maintained. Based on this intuition, we leverage multi-view stereo (MVS) algorithm identify these "image fragments" and generate geometry priors for 3D Gaussians.
Specifically, employ deep MVS method 48 generate depth maps for all views. A geomet- ric consistency filtering process 37 is leveraged to generate masks M, masking the multi-view want to identify. The masked
consistent areas, which essentially the "image fragments" depth maps are then mapped to 3D space as dense point clouds, which are used to initialize 3D Gaussians physically accurate positions. To find more multi-view consistent "image fragments," we propose suppressing the obstructions in training images by employing our proposed method. Specifically, we remove obstructions from input images to obtain transmission fţ. For areas blocked by occlusions(where & has large value), inpaint the content with 1/-

---

PSNR:32.7 75
(a) Reference
(b) 3DGS
PSNRC35.83
(c) Transmission (ours)
(d) Composed (ours)
Figure 5: Comparisons with 3DGS on novel view synthesis. Because the obstructions violate multi- view consistency, the performance of 3DGS degrades significantly, resulting in artifacts and blurry renderings (highlighted by red arrows). In contrast, our method not only faithfully synthesizes novel view renderings but also renders transmission with fine details, exhibiting improvement of 3.05dB in terms of PSNR.
1 == 110-1)1(11-00
if <T
otherwise
Here 1 is the ju, input image. 1g and 1f are synthesized by trained IOM and 3DGS, respectively. Eq. 7 is derived from Eq. 4
We use 0.5 for the threshold 7 in all the experiments.
3.5 Implementation Details
We develop our method based on 3DGS. We borrow multi-resolution hash encoding and fast MLP implementation from tiny-cuda-nn (31) to build IOM. We choose PatchMatchNet [48) the MVS method in G3E, follow previous driving scenes reconstruction works [4 10 to separately model the sky areas. The final loss function is:
Cpho + Ai + X2Copacity
(8)
pho is the same as in standard 3DGS [9). Laky is borrowed from UCNeRF 10]. We use L1 loss Lopacity ((.,)) [] u)]li to regularize the opacity field, where (u, v) are the image coordinates. This opacity loss encourages the opacity map to have the minimum areas that could satisfy the optimization. This design is based on the prior knowledge that opaque objects typically occupy only small portions of the windshield. We use 0.001 for both λι and A2. We run all the experiments with an A GPU, Each scene contains approximately 300 images in our evaluation datasets. Combined training of the 3DGS and IOM for 30k iterations with Adam optimizer takes about 30 minutes. It takes 40 minutes in total to evaluate MVS and run geometry filtering.
4 Experiments
4.1 Datasets
BDD100K. evaluate the performance of our method and baselines, we adopt BDD100K [59] for evaluation. This dataset contains 8 scenes that are from dash cam videos captured in daily life. They contain common obstructions, such as reflections, mobile phone holders, stickers and stains. Evaluation this dataset reflects performance on real life dash cam videos.
7

---

Table I: Evaluation of novel view synthesis on BDDI00K and DCVR. We indicate the best and second best with bold and underlined respectively. Our method consistently outperforms state-of offe-oofttue sitie--ffth-aatt t-thh-rrt methods both datasets and all the evaluation metrics.
BDD100K
DCVR SSIM f
Method
PSNR t
SSIM f
LPIPS +
PSNR f
LPIPS +
FPS f
NeRF-W 229
22.58
0.708
0.395
0.18
ZipNeRF
27.89
0.875
0.176
24.41
0.786
0.228
0.27
GaussianPro Pro 19
27.75
0.894
0.192
23.71
0.770
0.270
210
3DGS
19
28.02
0.897
0.188
23.73
0.783
0.248
155
DCGaussian (Ours)
29.44
0.914
0.143
24.74
0.822
0.202
120
DCVR. To further evaluate the performance of our method on strong reflection conditions, we established DCVR (Dash Cam Videos with Reflection) dataset. This dataset contains 10 dash videos we collect. The original videos are undistorted [5] to ease the structure- ceetuure-foo---ee from-motion algorithm. We utilize the popular tools COLMAP 37 and HLoc [35 [6] to estimate the camera parameters.
In both datasets, each sequence consists of approximately 300 frames, extracted from 10-second videos at a frame rate of 30 Hz. For scenes containing car hoods, the lower parts of the images are removed during preprocessing. Seven out of eighteen scenes in two datasets involve car turns, introducing diverse illumination changes. Additional details and visual results provided in the appendix.
4.2 Baselines
We choose 3DGS our baseline 19) We also compare our method with other state-of-the-art methods ZiR-NeRF 4. NeRF-W 29 and GaussianPro (9. We use the unofficial implementations of Zip-NeRF and NeRF-w To evaluate the performance of novel view synthesis, following common settings BI, we select one of every eight images as testing images and the remaining ones for training. Since our method is also designed for image decomposition, also compare method with state-of-the-art obstruction removal methods, including DSRNet [17), NIR [32 and Liu al. (27).
4.3 Quantitative Results
For the quantitative evaluation, we conduct comparison with baselines on both BDDI00K and DCVR. We apply the three widely-used metrics for evaluation, i.e., PSNR, SSIM 49 and LPIPS [2] The results are shown in Tab. [ The consistent superior performance of our method shows the efficacy of the proposed modules. Though GaussianPro and Zip-NeRF achieve great performance in obstruction-free scenes with their progressive propagation strategy and anti-aliasing mechanism, without separate obstruction modeling. they camnot handle the obstructions corrupted dash cam videos. NeRF-W is designed to handle illumination and content difference between images taken at different times but still cannot handle obstructions on the windshield. We show more visual results in the uppendix. DC-Gaussiaa achieves 120 fps resolution of 1920x 1080 RTX 3090 GPU. Although ours is slightly slower than 3DGS, the speed still enables real-time rendering, which is crucial for applications such as autonomous driving simulators.
4.4
Qualitative Results
Novel view synthesis. We show the novel view synthesis results in Fig. 5 Without a proper representation for obstructions, 3DGS can hardly synthesize high-quality obstructions. Moreover, its wrong geometry also results blurry renderings and artifacts on the road surface. In contrast, our method effectively tackles the ambiguity between obstructions and transmissions and synthesizes both components with high-fidelity. We provide depth map in the appendix.
Obstruction removal. We show the obstruction removal results in Fig. 6 The single image reflection removal method (e) only marginally suppresses reflections. Multi-image iilii-iimge layer separation methods
"https///gthhhh.com// com/Sul "https//githuu..oom//wweee

---

(a) Reference
(b) Transmission (ours)
(c) Obstruction (ours)
(d) NIR
(c) DSRNet
n Liu ct al.
(a) Reference
(b) Transmission (ours)
(c) Obstruction (ours)
(d) NIR
(e) DSRNet
(1 Liu et el.
Figure 6: Comparisons with the image- based reflection removal methods on removing reflections. (a) Reference a frame in a dash cam video. (b) and (c) are transmission and obstruction decomposed by our method. (d), (c), and (f) are results from previous obstruction removal methods, NIR (32). DSRNet [17 and Liu et al. [27], which are not effective in this scenario. In comparison, our method decomposes the image and synthesizes (b) transmission and (c) obstruction with high fidelity.
Table 2: Ablations studies on DCVR. Metrics are calculated on obstruction influenced areas.
NOM
AD
LIM
G3E
PSNR 1
SSIM t
LPIPS 4
X
X
23.99
0.738
0,287
X
X
X
25.21
0.776
0.252
J
X
x
25.65
0.791
0.236
V
x
25.90
0.798
0.229
V
26.30
0.814
0.210
(d)(f) struggle with accurate optical flow estimation, resulting in blurry outputs. In comparison, our method models reflections (c) with high quality and retains fine details in transmission (b). These visual results demonstrate our method's potential in reconstructing obstruction-free driving scenes from dash cam videos.
4.5 Ablation Study
We conduct extensive ablation studies on DCVR to explore the impact of each module in DC-Gaussian. Quantitative results are shown in Table 2 To assess the efficacy of global-shared hash encoding, evaluate 3DGS with Naive Obstruction Module (NOM), where latent features are directly decoded by an MLP to generate RGB. NOM shows sigmificant improvement over the baseline, demonstrating that global-share hash encoding effectively leverages the static prior of obstructions in dash cam videos. The adaptive image decomposition (AD) strategy further enhances results by effectively modeling occlusions, as shown in Fig. 7 Additionally, L.IM improves performance by capturing intensity changes reflections. Finally, incorporating geometry priors into 3D Gaussians with G3E effectively suppresses artifacts on the road and reveals sharper details, illustrated in Fig.

---

Reference
Transmission w/o Ø
Transmission w/
Learned Φ
Figure 7: Ablation study about the Learnable opacity map Φ. Incorporating the opacity map allows our method to accurately identify the positions of opaque objects, enhancing physical simulation and improving view synthesis and obstruction removal. Without the opacity map, severe artifacts appear.
G3E
(b) G3E
Figure 8: Ablation study on G3E module. G3E helps suppress artifacts and reveal sharper details.
Conclusions
In conclusion, we propose DC-Gaussian, which effectively addresses the challenges of extending 3D Gaussian Splatting to dash cam videos for the first time. The proposed Adaptive Image Decom- position module enables unified modeling of reflections and occlusions. To handle the reflections and occlusions under challenging lighting conditions, we introduce Illumination-awee aware Obstruction modeling. Additionally, we employ a coometry-Guided Gaussian Enhancement strategy further improve rendering quality. Experiments on BDD100K and DCVR demonstrate significant improve- ments in rendering quality and image decomposition, setting a new benchmark for neural rendering with dash cam videos.
Limitations and future work Currently, DC-Gaussian has only been evaluated single-sequence videos. However, considering the vast amount of dash cam footage available, extending DC-Gaussian multi-sequence video setting and leveraging dense view images to achieve more pleasing results would be promising direction for future research. In addition, Our method is not specifically designed to improve performance on dynamic scenes. We provide additional experimental results in the appendix. The results demonstrate that dynamic objects do not significantly impact the performance of obstruction removal. When dynamic objects move at slow speed, our method also presents reasonable results. We plan to incorporate techniques (53] for dynamic objects modeling into our method in future research enable robust dynamic modeling.
Acknowledgments and Disclosure of Funding
The authors acknowledge Advanced Research Computing at Virginia Tech for providing computa- tional resources and technical support that have contributed to the results reported within this paper. URL: https://arc.vt.edd vt.edu/
10

---

References
[1] Nikolaos Arvanitopoulos, Radhakrishna Achanta, and Sabine Susstrunk. Single image reflection suppression. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4498-4506, 2017.
[2] Connelly Barnes, Eli Shechtman, Adam Finkelstein, and Dan B Goldman. Patchmatch: A ran- domized correspondence algorithm for structural image editing. ACM Trans. Graph., 28(3):24, 2009.
[3] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin- and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3855-5864, 2021.
[4] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19697-19705, 2023.
[5] G. Bradski. The OpenCV Library. Dr. Dobb's 's Journal of Software Tools, 2000.
[6] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pages 1 11621-11631. 2020.
[7] Rohan Chandra, Xijun Wang, Mridul Mahajan, Rahul Kala, Rishitha Palugulla, Chandrababu Naidu, Alok Jain, and Dinesh Manocha. Meteor: dense, heterogeneous, and unstructured traffic dataset with rare behaviors. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 9169-9175. IEEE, 2023.
[8] Zhengping Che, Guangyu Li. Tracy Li, Bo Jiang, Xuefeng Shi, Xinsheng Zhang, Ying Lu, Guobin Wu, Yan Liu, and Jieping Ye. D²-city: large-scale dashcam video dataset of diverse traffic scenarios. arXiv preprint arXiv:1904.01199 2019.
[9] Kai Cheng, Xiaoxiao Long, Kaizhi Yang, Yao Yao, Wei Yin, Yuexin Ma, Wenping Wang, and Xuejin Chen. Gaussianpro: 3d gaussian splatting with progressive propagation. arXiv preprint arXiv:2402. 14650, 2024.
[10] Kai Cheng, Xiaoxiao Long, Wei Yin, Jin Wang, Zhiqiang Wu, Yuexin Ma, Kaixuan Wang, Xiaozhi Chen, and Xuejin Chen. Uc-nerf: Neural radiance field for under-calibrated multi-view cameras. In The Twelfth International Conference Learning Representattonss 2023.
[11] Ilya Chugunov, David Shustin, Ruyu Yan, Chenyang Lei, and Felix Heide. Neural spline fields for burst image fusion and layer separation. arXiv preprint arXiv:2312. 14235,, 2023.
[12] Qingnan Fan, Jiaolong Yang, Gang Hua, Baoquan Chen, and David Wipf. A generic deep architecture for single image reflection removal and image smoothing. In Proceedings of the IEEE International Conference on Computer Vision, pages 3238-32222 2047,,27,200 2017.
[13] Hany Farid and Edward H Adelson. Separating reflections and lighting using independent components analysis. In Proceedings. 1999 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (Cat. No PR00149), volume pages 262-267. IEEE, 1999.
[14] Xiao Fu, Shangzhan Zhang, Tianrun Chen, Yichong Lu, Lanyun Zhu, Xiaowei Zhou, An- dreas Geiger, and Yiyi Liao. Panoptic nerf: 3d-to-2d label transfer for panoptic urban scene segmentation. In 2022 International Conference on 3D Vision (3DV), pages 1-11. IEEE, 2022.
[15] Yosef Gandelsman, Assaf Shocher, and Michal Irani. ddoubleddip:: unsupervised image decomposition via coupled deep-image-priorr. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[16] Yuan-Chen Guo, Di Kang, Linchao Bao, Yu He, and Song-Hai Zhang. Nerfren: Neural radiance fields with reflections. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 18409-18418. 2022.

---

[17] Qiming Hu and Xiaojie Guo. Single image reflection separation via component synergy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13138- 13147, 2023.
[18] Yan Huang, Yuhui Quan, Yong Xu, Ruotao Xu, and Hui Ji. Removing reflection from a single image with ghosting effect. IEKE Transactions Computational Imaging, 6:34 2019.
[19] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(44)1-14, 2023.
[20] Chenyang Lei and Qifeng Chen. Robust reflection removal with reflection-free free flash-only cues. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 14811-14820 2021.
[21] Chenyang Lei, Xuhua Huang, Mengdi Zhang, Qiong Yan, Wenxiu Sun, and Qifeng Chen. Polarized reflection removal with perfect alignment in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1750-1758, 2020.
[22] Chenyang Lei, Xudong Jiang, and Qifeng Chen. Robust reflection removal with flash-only cues in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2023.
[23] Anat Levin and Yair Weiss. User assisted separation of reflections from single image using sparsity prior. IEEE Transactions on Pattern Analysis and Machine Intelligence, 19(9):6647- 1654, 2007.
[24] Yu Li and Michael S Brown. Single image layer separation using relative smoothness. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2752- 2759, 2014.
[25] Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3292-3311 2022.
[26] Philipp Lindenberger, Paul-Edouard Sarlin, Viktor Larsson, and Mare Pollefeys. Pixel-Perfect Structureffrom-Motiio with Featuremetric Refinement. In ICCV, 2021.
[27] Yu-Lun Liu, Wei-Sheng Lai, Ming-Hsuan Yang. Yung-Yu Yu Chuang, and Jia-Bin Huang. Learning see through obstructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 14215-14224, 2020.
[28] Youwei Lyu, Zhaopeng Cui, Li, Marc Pollefeys, and Boxin Shi. Reflection separation using a pair of unpolarized and polarized images. Advances in neural information processing systems, 32, 2019.
[29] Ricardo Martin-Brualle Noha Radwan, Mehdi SM Sajjadi, Jonathan T Barron, Alexey Doso- vitskiy, and Daniel Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pages 7210-7219, 2021.
[30] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoor- thi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65())999-006 2021.
[31] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM transactions on graphics (T0G),41(4):. 15, 2022.
[32] Seonghyeon Nam, Marcus A Brubaker, and Michael S Brown. Neural image representations for multi-image fusion and layer separation. European conference on computer vision, pages 216 216-232. Springer, 2022.
12

---

[33] Jiaxiong Qiu, Peng-Tat Tao Jiang, Yifan Zhu, Ze-Xin Yin, Ming-Ming Cheng, and Bo Ren. Looking through the glass: Neural surface reconstruction against high specular reflections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 20823-20833, 2023.
[34] Grand View Research. Dashboard camera market size, share & trends analysis report by technology (basic, advanced, smart), by product, by video quality, by application. by distribution channel, by region, and segment forecasts, 2024 2030, 2023. Accessed: 2024-05-16.
[35] Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In CVPR, 2019.
[36] Yoav Y Schechner, Joseph Shamir, and Nahum Kiryati. Polarization-bast decorrelation of transparent layers: The inclination angle of an invisible surface. In Proceedings of the seventh IEEE international conference on computer vision, volume 2, pages 814-812 819. IEEE, 1999.
[37] Johannes L Schönberger, Enliang Zheng. Jan-Michael Frahm, and Mare Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part
III 14, pages 501-518. Springer, 2016.
[38] YiChang Shih, Dilip Krishnan, Fredo Durand, and William Freeman. Reflection removal using ghosting cues. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3193-3201, 2015.
[39] Christian Simon and In Kyu Park. Reflection removal for in-vehicle black box videos. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4231-4239, 2015.
[40] Chao Sun, Shuaicheng Liu, Taotao Yang, Bing Zeng, Zhengning Wang, and Guanghui Liu. Automatic reflection removal using gradient intensity and motion cues. Proceedings of the 24th ACM international conference on Multimedia, 2016.
[41] Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. Pwe-net: Cnns for optical flow using pyramid, warping, and cost volume. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8934 9934-8943, 2017.
[42] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaík, Paul Tsui, James Guo. Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2446 2454, 2020.
[43] Richard Szeliski, Shai Avidan, and Padmanabhan Anandan. Layer extraction from multiple images containing reflections and transparency. In Proceedings IEEE Conference on Computer Vision and Pattern Recognition. CVPR 2000 (Cat. No. PR00662), volume 1, pages 246-253. IEEE, 2000.
[44] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar, Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248 8258, 2022.
[45] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construc- tion of lcage-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12922-1293.. 2022.
[46] Haithem Turki, Jason Y Zhang, Francesco Ferroni, and Deva Ramanan. Suds: Scalable urban dynamic scenes. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12375-12385, 2023.
[47] Renjie Wan, Boxin Shi, Haoliang Li, Yuchen Hong, Ling- Yu Duan, and Alex C Kot. Bench- marking single-image reflection removal algorithms. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 4542):1424-141,, 2022.
13

---

[48] Fangjinhua Wang, Silvano Galliani, Christoph Vogel, Pablo Speciale, and Mare Pollefeys. Patchmatchnet: Learned multi- patchmatch stereo. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14194-14203, 2021.
[49] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, 13(4);600- 612, 2004.
[50] Kaixuan Wei, Jiaolong Yang, Ying Fu, David Wipf, and Hua Huang. Single image reflection removal exploiting misaligned training data and network enhancements. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2019,
[51] Zirui Wu, Tianyu Liu, Liyi Luo, Zhide Zhong, Jianteng Chen, Hongmin Xiao, Chao Hou, Haozhe Lou, Yuantao Chen, Runyi Yang. et al. Mars: An instance-aware, modular and realistic simulator for autonomous driving. In CAAI International Conference Artificial Intelligence, pages 3-1 Springer, 2023.
[52] Ziyang Xie, Junge Zhang, Wenye Li, Feihu Zhang, and Li Zhang. S-nerf: Neural radiance fields for street views. arXiv preprint wrXiv:2303.00749 2023.
[53] Yunzhi Yan, Haotong Lin, Chenxu Zhou, Weijie Wang, Haiyang Sun, Kun Zhan, Xianpeng Lang, Xiaowei Zhou, and Sida Peng. Street gaussians for modeling dynamic urban scenes. arXiv preprint 2rXiv:2401.01333, 2024.
[54] Jiawei Yang, Boris Ivanovic, Or Litany, Xinshuo Weng, Seung Wook Kim, Boyi Li, Tong Che, Danfei Xu, Sanja Fidler, Marco Pavone, et al. Emernerf: Emergent spatial-temporal scene decomposition via self-supervision. arXiv preprint arXiv:2311.02077, 2023.
[55] Jie Yang. Dong Gong, Lingqiao Liu, and Qinfeng Shi. Seeing deeply and bidirectionally: A deep learning approach for single image reflection removal. In Proceedings of the european conference on computer vision (ECCV), pages 654 669, 2018.
[56) Yang Yang, Wenye Ma, Yin Zheng, Jian-Feng Cai, and Weiyu Xu. Fast single image reflection suppression via convex optimization. In Proceedings of the JEEE/CVF conference on computer vision and pattern recognition, pages 8141-8149, 2019.
[57] Ze Yang, Yun Chen, Jingkang Wang. Sivabalan Manivasagam, Wei-Chiu Ma, Anqi Joyce Yang, and Raquel Urtasun. Unisim: A neural closed-loop sensor simulator. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1389-1399, 2023.
[58] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. De- formable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.1311 2023.
[59] Fisher Yu, Haofeng Chen, Xin Wang, Wenqi Xian, Yingying Chen, Fangchen Liu, Vashisht Madhavan, and Trevor Darrell. Bdd100k: A diverse driving dataset for heterogeneous mul- titask learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2636-2645 2020.
[60] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402-416, 226,416,222 2018.
[61] Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492, 2020.
[62] Richard Zhang, Phillip Isola, Alexei Efros, Eli Shechtman, and Oliver Wang. The unrea- sonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595, 2018
[63] Xiaoshuai Zhang, Abhijit Kundu, Thomas Funkhouser, Leonidas Guibas, Hao Su, and Kyle Genova. Nerflets: Local radiance fields for efficient structure-aware 3d scene representation from 2d supervision. Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 8274-8284, 2023.
14

---

[64] Xuaner Zhang, Ren Ng, and Qifeng Chen. Single image reflection separation with perceptual losses. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 4786-4794, 2018 8.
[65] MI Zhenxing and Dan Xu. Switch-nerf: Learning scene decomposition with mixture of experts for large-scale neural radiance fields. In The Eleventh International Conference on Learning Representations, 2022.
[66] Hongyu Zhou, Jiahao Shao, Lu Xu, Dongfeng Bai, Weichao Qiu, Bingbing Liu, Yue Wang, Andreas Geiger, and Yiyi Liao. Hugs: Holistic urban 3d scene understanding via gaussian splatting. arXiv preprint arXiv:2403. 12722,, 2024.
[67] Xiaoyu Zhou, Zhiwei Lin, Xiaojun Shan, Yongtao Wang, Deqing Sun, and Ming-Hsuan Yang. Drivinggaussian: Composite gaussian splatting for surrounding dynamic autonomous driving scenes. arXiv preprint arX1v:2312.07720, 2023.
[68] Chengxuan Zhu, Renjie Wan, and Boxin Shi. Neural transmitted radiance fields. Advances in Neural Information Processing Systems, 55:38994-39006, 2022.
[69] Chengxuan Zhu, Renjie Wan, Yunkai Tang, and Boxin Shi. Occlusion-free scene recovery via neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10722-20733 2023.
[70] Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa splatting. IEEE Transactions on Visualization and Computer Graphics, 8(3):223-238, 2002.
15

---

A
Appendix supplemental material
PSNR8-33.52
a Zin-Nel
(b) 3DCS
( Ours
Composed
Depth map
Figure 9: Comparisons with 3DGS and Zip-NeRF on novel view synthesis show that obstructions violate multi-view consistency, leading to erroneous geometry in 3DGS and Zip-NeRF, evident in the depth maps. This results blurry renderings and artifacts. contrast, our method effectively addresses the ambiguity introduced by obstructions and learns physically reasonable geometry, achieving renderings with fine details.
(a. Reference
b Obstruction
(c) Depth map
(d) Transmission
Figure 10: We evaluate NeRFRen 16 on curated dataset. The suboptimal results of NeRFRen are caused by two factors. First, its obstruction modeling cannot address the ambiguity between obstructions and transmission, leading to failure in image decomposition. Second, its backbone, NeRF, cannot handle large-scale driving scenes, resulting in blurry outputs.
16

---

Table 3: Ablation threshold used in Eq.
Our results not sensitive to the choice of
PSNR f
SSIM t
LPIPS
0.3
26.27
0.814
0.210
0.5
26.30
0.814
0.210
0.7
26.28
0.814
0.210
Table 4: We first DSRNet [7 to remove reflections from the input images, and then train and evaluate 3DGS [19] on these images. The results show that due to the insufficiency of the reflection removal, novel view synthesis performance cannot be improved in this way.
Method
BDD100K
DCVR
PSNR
SSIM 1
LPIPS
PSNR 1
SSIM f
LPIPS
3DGS 3DGS DSRNet
28.02
0.897
0.188
23.73
0.783
0.248
27.99
0.898
0.188
23.72
0.783
0.248
Figure 11: Trajectories of turning cars, which result in diverse illumination changes.
Ground Truth
Composed
Obstructions
SCHOOL
SCHOOL
Figure 12: Visual results on dynamic scenes. All three scenes demonstrate that dynamic objects do not significantly impact the decomposition of obstructions. Our method achieves good performance in the scene shown in the first row, where the dynamic objects are moving slowly. However, in the second and third rows, where the dynamic objects are moving at higher speeds, our method shows suboptimal performance.
Figure 13: Nerf-in-ther the-inttthwwldd fails to separate obstructions from the images. None of the obstructions accurately represented in the transient image.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We show our method's performance novel view synthesis and obstruction removal, reflecting our contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of this paper in conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

The paper does not present any theoretical results requiring assumptions or proofs, making this criterion not applicable.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We cite used techniques and introduce method in details.
Guidelines:
The answer NA means that the paper does include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We will submit our code and data to github.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//guiii wwwticccguie//eeieeeeeeii CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We document all the training details in the implementation details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
It's too compute intensive to do so.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the GPU we used implementation details.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Question:
Answer: [Yes]
Justification:
We understand and respect the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: We discuss the positive societal impacts; however, we do not foresee any potential negative impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, the creators or original owners of the assets used in the paper are properly credited, and the license and terms of use are explicitly mentioned and properly respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
22

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide information about the dataset
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work doesn't involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Our work doesn't involve human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "7K7UWJqr3u", "reviewer": "Reviewer_si9q", "summary": "This paper focuses on using dash cam videos for 3D Gaussian Splatting-based outdoor scene reconstruction. To address challenges such as reflections and occlusions on windshields, DC-Gaussian introduces an adaptive image decomposition module to model these effects in a unified manner. Additionally, an illumination-aware obstruction modeling technique is designed to manage reflections and occlusions under varying lighting conditions. Finally, the authors employ a geometry-guided Gaussian enhancement strategy to improve rendering details by incorporating additional geometry priors. Extensive experiments on self-captured and public dash cam videos verify the effectiveness of the proposed method.", "questions": "See weaknesses.", "limitations": "The authors have adequately addressed the limitations and the potential negative societal impact of their work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) The proposed method exhibits promising 3DGS-based reconstruction performances on some self-captured and public dash cam videos. \n\n(2) In general, this manuscript is well-structured and the writing is clear. The authors have effectively organized their ideas, making the content easy to follow and understand.", "weaknesses": "(1) Limited Practicality: My main concern is the practical value of this work. Dash cam videos generally provide single-view sequences, and these sparse views would result in the learned 3DGS suffering from limited novel-view synthesis quality. Additionally, the inherent limitations of dash cam videos, such as reflections and occlusions on the windshields, would further degrade rendering quality despite the authors’ efforts to mitigate these issues with developed modules. Therefore, I find it hard to believe that people would choose dash cam videos for high-quality 3D scene reconstruction in real-world applications.\n\n(2) The authors introduce global-shared hash encoding and MLPs for illumination-aware obstruction modeling. The reviewer is curious about the running speed compared to the baseline method, 3DGS. Please provide a comparison of the inference times.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Concern about the practical value of using dash cam videos for Novel View Synthesis**\n\nDash cam videos have unique values for autonomous driving. Dash cam videos deeply reflect the diversity and complexity of real-world traffic scenarios. They are used to provide large-scale, diverse driving video datasets in a crowd-sourced manner [1]. Dash cam videos also offer important data sources about multi-agent driving behaviors [2] and evaluating the robustness of algorithms under visual hazards [6]. BDD100K [1], a large scale diverse driving videos dataset built with dash cam videos, has been cited more than 2000 times. It has been widely used \nin many important vision tasks, such as Object Detection, Semantic Instance Segmentation [8], and Multiple Object Tracking and Segmentation [7], etc. This fact shows the huge interests of vision community in dash camera studies.\n\nAlthough most dash cam videos are monocular and prone to obstructions, we believe studying how to leverage the important data source is quite important to many vision tasks, such as novel view synthesis and 3D reconstruction.\nMoreover, our work is an attempt to tackle obstruction removal from dash cam videos, making it possible to achieve high-quality NVS from dash cam videos. It will provide huge 3D information for autonomous driving.\n\n[1] Yu F, Chen H, Wang X, et al. Bdd100k: A diverse driving dataset for heterogeneous multitask learning. CVPR, 2020.\n\n[2] Chandra R, Wang X, Mahajan M, et al. Meteor: A dense, heterogeneous, and unstructured traffic dataset with rare behaviors. ICRA, 2023.\n\n[3] Grand View Research. Dashboard camera market size, share \\& trends analysis report by technology (basic, advanced, smart), by product, by video quality, by application, by distribution channel, by region, and segment forecasts, 2024 - 2030, 2023. Accessed: 2024-05-16.\n\n[4] Martin-Brualla, Ricardo, et al. \"Nerf in the wild: Neural radiance fields for unconstrained photo collections.\" CVPR, 2021.\n\n[5] Gao, Chen, et al. \"Dynamic view synthesis from dynamic monocular video.\" ICCV, 2021.\n\n[6] Oliver Zendel, Katrin Honauer, Markus Murschitz, Daniel Steininger, and Gustavo Fernandez Dominguez. Wilddash-creating hazard-aware benchmarks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 402–416, 2018.\n\n[7] Luiten, Jonathon, et al. \"Hota: A higher order metric for evaluating multi-object tracking.\" International journal of computer vision 129 (2021): 548-578.\n\n[8] Yan, Bin, et al. \"Universal instance perception as object discovery and retrieval.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n**2) Rendering speed.** This question is answered in the global rebuttal response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "je0LQPUSVh", "reviewer": "Reviewer_N7Lh", "summary": "This paper deals with the problem of novel view synthesis in outdoor scenes captured with a dashcam. The authors develop a 3D Gaussian Splatting based method that is robust to common obstructions observed in dashcam videos, mainly due to the way these videos are captured: Mobile-phone holders, reflections and windshield stains create artifacts in the reconstruction as they are not natively modeled by 3DGS and cannot be reconstructed as part of the static 3D scene as they move with the ego-vehicle. The authors propose a learnable opacity map that decomposes the images into obstructions and real scene and a lighting modulation module that adapts obstructions over for each view based on varying illumination across views. These are implemented with a 2D hashgrid and MLP heads. They can then use the opacity map to blend the obstruction and real scene for rendering each view. The authors also use an MVS method with geometric consistency filtering to initialize the 3D Gaussian positions.", "questions": "It would be great if the authors can clarify my concerns w.r.t. the loss details, G3E and may provide more insights on the choice of baselines.", "limitations": "The limitation discussion is short and does not point to meaningful failure cases.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed decomposition scheme is effective in separating obstruction from real scene as illustrated in Fig. 4/5/6.\n- The accompanying video results are impressive\n- The different modules are clearly ablated in Tab. 2", "weaknesses": "- There is no description of $\\mathcal{L}_{opacity}$, but without this it is not possible to judge whether the learned opacity map is learned in a self supervised manner or if additional labels are needed.\n- The task setup is similar in spirit to NeRF-in-the-wild. Therefore, to judge the importance of the proposed design opposed to generic in-the-wild methods (NeRF-W,  not tailored towards dashcam videos, it would be important to understand how such methods compare on the benchmarks the authors use. However, the only baselines the authors compare to are ZipNeRF, 3DGS and GaussianPro, all of which are not designed for this type of captures.\n- As far as I understand, G3E consists of running PatchMatchNet with a standard geometry consistency filter. There seems to be no innovation in that to me. What is Fig. 8 comparing exactly? Random initialization vs MVS or SfM vs MVS initialization?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Details of $\\boldsymbol{\\mathcal{L}_{opacity}}$.** The formulation of opacity loss is shown in the global response. The opacity map is learned in a self-supervised way without relying on additional labels.\n\n**2) Why only compare with general Novel View Synthesis methods?**\n\nAs the first work addressing the novel task of NVS for dash cam videos, there are no prior works specifically designed for this task. As a result, we compare ours with several SOTA methods.\nNeRF-W is designed for NVS using images captured with different cameras and at different times but still cannot handle obstructions like reflections on the windshield. We evaluate NeRF-W on the dash cam videos, and as shown in the table below, our method significantly outperforms NeRF-W in both synthesis quality and rendering speed.\n\nTable 2. Comparison with NeRF-W. Tested on BDD100K\n| Method             | fps $\\uparrow$ | PSNR $\\uparrow$ | SSIM $\\uparrow$ | LPIPS $\\downarrow$ |\n|--------------------|:----------------:|:-----------------:|:-----------------:|:--------------------:|\n| Nerf in the Wild   |       0.18       |       22.58       |       0.708       |         0.395        |\n| DC-Gaussian (Ours) |        120       |       29.44       |       0.914       |         0.143        |\n\nFurthermore, as illustrated in Figure 4 in the attached pdf, although NeRF-W is designed to handle illumination variance and transient objects, it fails to separate obstructions from the images. None of the obstructions are accurately represented in the transient image. This suboptimal result occurs because the obstructions move with the camera rather than being transient parts of the images. The reflections and transmissions are intertwined in a complex manner that NeRF-W's design cannot effectively address, unlike our method. \n\n\n**3) Novelty of G3E.**\nWe use the standard geometry consistency filter from PatchMatchNet to improve novel view synthesis.\nSome works also leverage MVS priors to help NVS task.\nHowever, simply applying PatchMatchNet to original input images does not yield optimal results. In our method, we first run DC-Gaussian without multi-view stereo (MVS) initialization. We then use the trained model to synthesize $\\hat{I}_t$, as explained in Equation 7. This strategy effectively suppresses obstructions in the original images and enhances overall performance. In Fig 8, we compare the results using MVS initialization (a w/ G3E) and that using SfM initialization (b w/o G3E), we will make it more clear in revised paper.\n\n**4) Limitations.** The limitations are discussed in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7NLbaZIVej", "reviewer": "Reviewer_zoS8", "summary": "This paper presents a new Gaussian Splatting novel view synthesis method specifically for in-vehicle dash cam videos containing various obstructions. To the challenges caused by obstructions, the proposed method separately represents the transmission and obstruction part of the camera capture. The transmission part is a 3DGS representation, and the obstruction part is modeled as a 2D representation with an opacity map and position-dependent illumination-aware neural model. The synthesized transmission images are also used to enhance the geometry prior for 3DGS training. The evaluation results show apparent improvements over previous methods.", "questions": "* I wonder how well the proposed method works for the reflective car hoods, which are also commonly seen from the dash cam.", "limitations": "Limitations are discussed at the end of the paper, but I don’t think it is adequate, see my comments in the above sections.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper has an interesting motivation for a neglected but meaningful task on driving scene reconstruction with the dash cam. The paper is well-written and easy to follow. The paper also gives adequate references to related work.\n2. The proposed solution looks reasonable and very effective on this specific task. I love to see how the authors leverage the key observations to design their solutions.\n3. The latent intensity modulation (LIM) is an interesting and smart design for handling varying illumination conditions on the road.", "weaknesses": "1. The proposed method is more like an improvement for a specific Gaussian Splatting application. The similar components in the proposed method can be found, in part, from other related work. Therefore, the novelty of this paper is moderate.\n\n2. In Sec. 3.2, the authors reformulate the composition equation to Eq. 4. This equation is a bit incorrect from the perspective of light transport. I feel the $(1 - \\phi) I_t + (\\phi) I_o $ would make more sense.\n\n3. In the experiments, it is still unclear about the evaluation datasets. The following key informations are missing in the paper:\n\n   * The number of scenes used and the number of frames per scene.\n   * Image quality, and resolutions (since dash cam might have worse image quality).\n   * Do the scenes contain dynamic objects?\n   * Is there any scene containing opaque occlusions such as a car hood?\n   * Do the drive sequences contain any turning to show more diverse illumination changes?      \n\n   I would suggest the authors show more visual results in the revised version.\n\nI am looking forward to the authors' responses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Moderate novelty because the similar components in the proposed method can be found, in part, from other related work.**\n\nWe appreciate if the reviewer could be more specific about the components. Then we can provide detailed explanations during the discussion period. \nWhile both 3DGS and hierarchical hash encoding are tools that have been developed and widely used in related works, the novelty of this paper lies in our effective design of a framework that utilizes physical priors to address this challenging task for the first time.\n\n**2) Correctness of equation 4.**\nThank you for pointing out this typo. The physically correct version of the composition equation should be:\n\n$$\n  I(u, v, j) = (1 - \\boldsymbol{\\phi})I_t(u,v,j) + (\\boldsymbol{\\phi}) I_o(u,v,j)\n$$\n\nWe will correct this typo in the revised version of the paper.\n\n**3) Details of the datasets.**\nThank you for bringing this up. We will include these details in the revised paper.\n\n- The curated BDD100K dataset contains 8 scenes, while the self-collected DCVR dataset includes 10 scenes. Each sequence consists of approximately 300 frames, extracted from 10-second videos at a frame rate of 30 Hz.\n- Modern dash cameras provide decent quality. The resolution of the images in BDD100K is around 1296x500. For the DCVR dataset, we used a 4K wide-angle camera (purchased on Amazon, selected based on high sales volume). The original resolution of these images is 3840x2160. We performed camera calibration and undistortion using a chessboard and the OpenCV library before running the SFM algorithm. To facilitate the training process, we downsampled the images 3 times to a resolution similar to that in BDD100K, aligning with popular datasets like KITTI-360 and BDD100K.\n- 4 out of 8 scenes in BDD100K contain moderate dynamic objects. No scenes in DCVR contain dynamic objects.\n- As shown in Fig. 7 in the paper, our dataset includes opaque occlusions. Our dataset does not contain car hoods, which we will discuss later.\n- 3 out of 8 scenes in BDD100K contain turning. 4 out of 10 scenes in DCVR contain turning. We visually depict the turning trajectories in Fig. 2 of the attached PDF. \n\n\n**4) How well the proposed method works for the reflective car hoods?**\nThank you for bringing up the topic of car hoods. While car hoods are indeed prone to strong reflections, they typically occupy only around 10\\% of the lower parts of the images, which do not contain much information about the street scenes. Additionally, car hoods are consistently located in the same position in each video. We addressed this issue by manually cropping out the areas containing the car hoods during the dataset curation process. We believe that this can still be done on a large scale, as segmenting car hoods is not a particularly difficult task.\n\n**5) Limitations.** Limitations are discussed in the global response.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v00S9PN9me", "reviewer": "Reviewer_QCPG", "summary": "This paper tries to reconstruct 3DGS (3D Gaussian Splatting) with Dash Cam Videos, which introduces obstructions on windshields. An adaptive image decomposition is applied to learn transmission images and obstruction images separately. The former is modeled with 3DGS in G3 Enhancement and are rendered into 2D space, while the latter is learned on 2D camera space conditioned by car positions in Illumination-aware Obstruction Modeling. Further, an opacity map is optimized to weigh these two images. This method surpasses Zip-NeRF, 3DGS and other runner-ups in public and self-collected datasets.", "questions": "1. How to calculate opacity loss in Eq.8?\n2. Are there any visualizations related to reconstructed 3DGS?\n3. How fast is the rendering?", "limitations": "As stated in this paper, DC-Gaussian has only been evaluated on single-sequence videos. I would also prefer the opacity map to be detailed modeled. However, considering the good performance, this trade-off on opacity is acceptable.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Originality: This paper addresses the issue faced in daily life where perfect capturing is hard to get. The setting of Dash Cam Videos is novel, and I'm happy to see the introduction of 3DGS to model such straight-ahead-captured data.\n\nQuality: The results seem good, with high scores on rendering and outstanding decomposition on transmission and obstruction.\n\nClarity: I believe the paper and diagrams are easy to read.\n\nSignificance: This may do help to autonomous driving considering its high-speed rendering.", "weaknesses": "1. The opacity map is modeled to be independent of car position, but in fact, according to Fresnel's law, opacity is also related to the light distribution, and therefore also related to the position.\n2. The two observations in 3.3 are too strong. This somehow weakens the generalization of the algorithm.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1) Why is the opacity map modeled to be independent of car position?**\n\nThank you for pointing out that, according to Fresnel's law, the opacity of the window should also depend on the car's position. \nHowever, in experiments, we found that most of occlusions are caused by the objects between dash camera and car window, such as mobile phone holders and stains, rather than the objects outside the car. Since these objects remains relatively still to cars, it is reasonable to design the opacity map to be independent of the car's position. Given the extremely ill-posed nature of this task, we chose to omit the dependence between window's opacity and car's position. \n\n**2) The two observations in 3.3 about the static motion prior and vary illuminations of obstructions are too strong.**\n\nWe acknowledge that the assumptions guiding our design are quite strong. However, this task is highly ill-posed, as evidenced by the suboptimal results of previous methods, such as some reflection removal models trained on paired datasets. These strong assumptions have been proved to be crucial for achieving satisfactory performance. \nWe will explore to relax these assumptions in the future research, enabling adaptation to a wider range of scenarios.\n\n**3) Details of $\\boldsymbol{\\mathcal{L}_{opacity}}$.** This question is answered in the global response.\n\n**4) Rendering speed.** This question is answered in the global rebuttal response.\n\n**5) Visualization of 3DGS.** We provide a visualization of the generated 3DGS in Figure 1 of the attached PDF. Due to the one-page limit of the attached document, please refer to our supplementary materials for additional visual results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are grateful to all reviewers for their insightful and constructive suggestions. We are glad that reviewers found: (1) The problem setting is novel (Reviewer QCPG) and meaningful (Reviewer zoS8); (2) The proposed method is interesting, smart (Reviewer zoS8), and effective (Reviewer zoS8, N7Lh); (3) The results look promising (Reviewer si9q), impressive (Reviewer N7Lh), good, and outstanding (Reviewer QCPG). Each comment from the reviewers has been replied to individually. Here we first respond to common questions.

**1) Rendering speed.** As shown in the table below, DC-Gaussian achieves 120 fps at a resolution of 1920x1080 on an RTX 3090 GPU. Although ours is slightly slower than 3DGS, the speed still enables real-time rendering, which is crucial for applications such as autonomous driving simulators.

Table 1. Rendering Speed Analysis. Tested on one NVIDIA RTX 3090 GPU with image resolution $1920\times1080$
| Method             | fps $\uparrow$ |
|--------------------|:----------------:|
| Nerf in the Wild   |       0.18       |
| ZipNeRF            |       0.27       |
| GaussianPro        |        210       |
| 3DGS               |   155  |
| DC-Gaussian (Ours) |        120       |

**2) Details of $\mathcal{L}_{opacity}$.**  The formulation of **$\mathcal{L}_{opacity}$**  is $\sum_{(u, v)}\Vert \phi(u, v) \Vert_{1}$, where $(u, v)$ are the image coordinates. This opacity loss encourages the opacity map to have the minimum areas that could satisfy the optimization. This design is based on the prior knowledge that opaque objects typically occupy only small portions of the windshield.

**3) Limitations.**  
Our work focuses on obstruction removal during Novel View Synthesis (NVS) for dash cam videos. Our method is not specifically designed to improve performance on multiple sequences or dynamic scenes. These limitations have been discussed in the paper. Here, we provide additional experimental results in Figure 3 of the attached PDF. The results demonstrate that dynamic objects do not significantly impact the performance of obstruction removal. When dynamic objects move at a quite slow speed, our method also presents reasonable results. Recent efforts on modeling dynamic objects with high speed, such as StreetGaussian [1], have achieved impressive results. We plan to incorporate these techniques into our method in future research to enable robust dynamic modeling.

[1] Yan, Yunzhi, et al. "Street gaussians for modeling dynamic urban scenes." ECCV, 2024.

Meta-review: As the Area Chair for submission 4323, I have thoroughly evaluated the reviewers' feedback and the authors' responses. The consensus leads me to recommend acceptance of this paper for NeurIPS 2024.

The paper addresses novel view synthesis with dash cam videos, a topic of growing importance but limited exploration. It introduces DC-Gaussian, an innovative method that effectively tackles unique challenges such as obstructions and variable lighting in dash cam footage. The experimental results are robust, showing clear improvements over existing methods like NeRF-W, especially in handling common dash cam obstructions. 

Reviewers initially expressed concerns about the practicality and the handling of dynamic scenes. However, the authors provided comprehensive rebuttals that demonstrated the method's utility and robustness, supplemented by recent literature and additional experimental validations.

he authors are encouraged to incorporate final feedback regarding dataset details and benchmarking in their final manuscript. The acceptance of this paper will foster further research in a promising direction.
Decision: Accept (poster)