Title: 4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization
Venue: NeurIPS 2024
Accepted: yes
Keywords: Dynamic Novel View Synthesis

4D Gaussian splatting

Sparse reconstruction

in-the-wild
Abstract: Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality.
We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. 
To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions.
This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. 
We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. 
To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. 
Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.

4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

Abstract

Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality. We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions. This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.

1 Introduction

Dynamic novel View Synthesis (DVS) aims to reconstruct dynamic scenes from captured videos and generate photorealistic frames for an arbitrary new combination of a viewpoint and a time step. This task has emerged as a vital research area in the 3D vision community with rapid advancements in augmented reality and virtual reality. Early DVS research primarily relied on neural radiance fields [29, 69, 10, 13, 11, 38, 39, 41, 5, 12, 50]. In contrast, more recent methods [61, 18, 31] extend 3D Gaussian Splatting [23] to account for the additional time dimension in dynamic scenes, and these techniques are referred to as 4D Gaussian Splatting.

Despite the recent success of 4D Gaussian Splatting models [61, 18, 31, 68], their applicability remains largely limited to controlled and purpose-built environments. Most existing models are developed and tested with multi-view video setups [29, 41]. While there are several methods tackling monocular video settings, these setups are still controlled and fall short of in-the-wild scenarios. For instance, [38, 69] maintain multi-view characteristics, where the camera captures a broad arc around a slow-moving object. Also, HyperNeRF [39] relies on unrealistic train-test splits, with both sets sampled from the same video trajectory, which renders the task closer to video interpolation than genuine novel view synthesis. In this paper, we focus for the first time on more natural, real-world monocular videos [14], where a single handheld camera moves around fast-moving objects.

In casually recorded monocular videos, which often lack sufficient multi-view information, 4D Gaussian Splatting algorithms tend to overfit the training frames in real-world scenarios. To address overfitting, recent regularization techniques [26, 7, 58, 67, 25, 36, 20] can be applied to provide

certain

Rendered image

Uncertainty map

Uncertain region

Diffusion

Gaussian primitives

: needing additional priors

Depth smoothness

Certain region

: preserved

LUA-diff

LUA-TV

additional priors for unseen views. However, these regularization techniques often involve a balancing issue: while they effectively improve novel view synthesis performance during testing, they inherently sacrifice the reconstruction accuracy of training images. Since both the reconstruction accuracy and the novel view synthesis quality are equally important in our target task, the trade-off caused by the naïve application of the regularization techniques is not desirable.

In this paper, we address this balancing issue with a simple yet effective solution: uncertainty-aware regularization. First, we quantify the uncertainty of each Gaussian primitive based on its contribution to rendering for training images. Then, a 2D uncertainty map is constructed for unseen views using an α -blending method. Regularization is selectively applied to uncertain regions, guided by the diffusion and depth smoothness priors, while low-uncertainty regions, where training data already provide sufficient reconstruction detail, are left unregularized, as illustrated in Figure 1. This approach results in a better balance between training and test performance, achieving good performance.

In real-world scenarios involving fast motions, especially in casually recorded videos, 4D Gaussian Splatting additionally faces considerable challenges with initialization. The algorithms based on Gaussian Splatting initialize Gaussian primitives using point clouds obtained by Structure from Motion (SfM) [47]. However, SfM struggles to reconstruct dynamic regions, particularly those with fast motion, often treating them as noise and leaving these areas without initialized primitives. Such an incomplete initialization disrupts training, causing primitives in static regions to be repeatedly cloned and split in an attempt to fill the dynamic areas. This can lead to an excessive number of primitives and, in some cases, out-of-memory issues. To address this limitation, we propose a dynamic region densification technique that initializes additional Gaussian primitives in dynamic regions.

We address the challenging problem of 4D reconstruction from an in-the-wild monocular video recorded casually with a handheld camera-a scenario that has been rarely explored. The main contributions of this paper are summarized as follows:

We propose an uncertainty quantification technique based on contribution to training image rendering and introduce adaptive regularization techniques based on the uncertainty map, which balances between novel view synthesis performance and training image reconstruction quality.

We address the issue of incomplete initialization in dynamic regions, emphasizing the importance of proper initialization in the training process of 4D Gaussian Splatting.

We demonstrate the effectiveness of our algorithm on casually recorded monocular videos, showing improvements over baselines. Additionally, we validate the applicability of our method in few-shot static scene reconstruction.

The rest of this paper is organized as follows. Section 2 reviews related work and Section 3 discusses the basic concepts of 4D Gaussian splatting, which builds upon 3D Gaussian Splatting by integrating

deformation strategies. The details of our approach are described in Section 4, followed by the presentation of experimental results in Section 5. Finally, we conclude this paper in Section 6.

2 Related Work

2.1 Dynamic Novel View Synthesis

In recent years, significant advances have been made in novel view synthesis [34, 6, 15, 60, 35, 23]. Initially focused on static scenes, novel view synthesis has shifted towards dynamic scenes through the integration of motion modeling, now referred to as Dynamic novel View Synthesis (DVS). Early approaches [13, 10, 38, 5, 12, 50, 11] are largely driven by Neural Radiance Fields (NeRF). Some studies [13, 10] capture dynamics implicitly through temporal inputs or latent codes, and other approaches [38, 5, 12, 50, 11] focus on training the canonical NeRF and its deformation fields.

The introduction of 3D Gaussian Splatting (3DGS) [23] has marked a paradigm shift in novel view synthesis, leading to the development of 4D Gaussian Splatting (4DGS) [68, 18, 61, 31] for DVS. These 4DGS methods deform canonical 3D Gaussian primitives over time using additional deformation networks, which can be based on MLPs, learnable control points, Hexplane [5, 61], or polynomial functions. While these models excel at reconstructing dynamic scenes in controlled environments [38, 69, 29, 41, 39], they face significant challenges when applied to casually recorded monocular videos, posing substantial hurdles for real-world applications.

2.2 Regularization Techniques in Sparse Reconstruction

Casually recorded monocular videos typically provide limited multi-view information, as they are typically captured with a single handheld camera that only exhibits gentle motion. Consequently, reconstructing dynamic scenes from these videos is often regarded as a form of sparse reconstruction due to the lack of multi-view data.

In the context of sparse reconstruction, various regularization techniques have been proposed to mitigate overfitting on limited training images [36, 25, 20, 48, 22, 4, 49, 53, 24, 63, 43, 8, 59, 55, 16, 65, 62, 2, 17, 16]. These approaches generally involve rendering images or depth maps for unseen views to provide additional priors. For instance, some methods leverage depth priors based on estimated depths for novel views [43, 8, 59, 55, 16, 72], while others incorporate color smoothness constraints to enhance these views [25, 36]. Building on the success of diffusion models [44], recent approach [62] starts to incorporate diffusion priors to produce more realistic images of novel views. While these methods effectively enhance novel view synthesis performance at test time, they inherently compromise the quality of training image reconstructions. Since both the reconstruction accuracy and the novel view synthesis quality are equality important in our target task, the trade-off caused by the naïve application of the regularization techniques is not desirable.

2.3 Uncertainty Quantifications in Novel View Synthesis

Uncertainty estimation in novel view synthesis has primarily been explored with Neural Radiance Fields (NeRF) [29]. Pioneering approaches [37, 51, 52] re-parameterize MLP networks in NeRF using Bayesian models to compute the uncertainty of network predictions. Inspired by InfoNeRF [25], which considers entropy along rays for few-shot NeRFs, some studies [70, 66, 27] quantify uncertainty using the entropy of density along rays from a novel view. Additionally, Density-aware NeRF Ensembles [56] measures uncertainty by examining the variance in RGB images produced by an ensemble of models.

In contrast, uncertainty quantification in Gaussian Splatting [23] remains largely underexplored, with only a few works addressing this issue. Savant et al. [46] incorporate variational inference into the rendering pipeline, but this approach increases learnable parameters. Similarly, FisherRF [21] quantifies the uncertainty of Gaussian primitives by aggregating the diagonal of the Hessian matrix of the log-likelihood function; however, it is not straightforward to obtain a scalar value of uncertainty from the Hessian matrix for each Gaussian primitive. Our approach, on the other hand, directly quantifies the observed information of each Gaussian primitive by aggregating their contributions to the reconstruction of training images.

Most existing works that utilize estimated uncertainty primarily focus on quantifying model predictions after training [46] or on active learning for next-view selection [21]. In contrast, our approach leverages estimated uncertainty for adaptive regularization during training, specifically targeting Gaussian Splatting in sparse reconstruction.

3 Preliminary: 4D Gaussian Splatting

This section briefly overviews 3D Gaussian splatting (3DGS) [23] and explains the deformation modeling in 4D Gaussian Splatting (4DGS) [68, 18, 61, 31] for dynamic scenes.

3.1 3D Gaussian Splatting

Gaussian primitive 3D Gaussian splatting has demonstrated real-time, state-of-the-art rendering quality on static scenes. It uses an explicit 3D scene representation consisting of a set of 3D Gaussian ellipsoids, denoted by Γ = { γ 1 , ..., γ K } . Each Gaussian primitive, γ k , is based on an unnormalized 3D Gaussian kernel, G k ( x ) , parameterized by µ k and Σ k as follows:

where µ k ∈ R 3 is the center position, Σ k ∈ R 3 × 3 is an anisotropic covariance matrix, and x ∈ R 3 is an arbitrary location in 3D space. The covariance matrix Σ k is valid only when positive semi-definite, which is challenging to enforce during optimization. To ensure this condition, we learn Σ k by decomposing it into two learnable components, a rotation matrix R k and a scaling matrix S k as follows:

In addition to the standard Gaussian parameters such as µ k , R k , and S k , the Gaussian primitive requires additional learnable parameters for its opacity, α k ∈ [0 , 1] , and feature, f k ∈ R d , which is typically represented by RGB colors or spherical harmonic (SH) coefficients. Thus, each Gaussian primitive is represented as γ k := ( µ k , R k , S k , α k , f k ) .

Differentiable rasterization Before rendering with the Gaussian primitives Γ on an image space, each 3D Gaussian kernel, G k ( x ; µ k , Σ k ) , is projected onto a 2D image space and forms a 2D Gaussian kernel, G π k ( r ; µ π k , Σ π k ) , where π : R 3 → R 2 denotes a projection from the world coordinate to an image space. In the projected Gaussian representation, r ∈ R 2 indicates a pixel location in an image, and the 2D mean µ π k ∈ R 2 and covariance Σ π k ∈ R 2 are given by

where J denotes the Jacobian of the affine approximation of the projective transformation, and W is the world-to-camera transform matrix. When rendering the primitives in Γ to a target camera, they are sorted by their depths with respect to the camera center. The color of a pixel r is then obtained by α -blending, which is given by

where ω π k ( r ) represents a relative contribution of each Gaussian primitive to pixel r and c ( f k , r ) is the color of a pixel r measured along the view direction. If a feature vector f k is based on spherical harmonics coefficients, the color is decoded from f k using the view direction associated with pixel r ; otherwise, the feature vector f k can be identical to the RGB color of the primitive. For more details, please refer to the original Gaussian Splatting paper [23]. Note that, following the α -blending procedure in 3DGS [23], ω π k ( r ) is given by

where α k G π k ( r ; µ π k , Σ π k ) is the opacity of the k th projected primitive at the junction with a ray corresponding to pixel r and ∏ k -1 j =1 ( · ) is the transmittance at the primitive γ k on pixel r , which measure how much light penetrates the preceding primitives along the ray.

3.2 Deformation Modeling in 4D Gaussian Splatting

To represent 4D scenes using Gaussian splatting, recent algorithms [68, 18, 61, 31] deform the 3D Gaussian primitives from their canonical states to a target state over time. The transformed position µ t , rotation R t , and scale S t at time t are given by

where the deformation functions φ µ ( · ) , φ r ( · ) , and φ s ( · ) can be various forms, including MLPs, learnable control points [18], Hexplane [5, 61], or polynomial functions. A deformed 3D Gaussian primitive at time t is represented as γ k ( t ) := ( µ t k , R t k , S t k , α k , f k ) . The projection onto a 2D space follows the same procedure as the static 3D Gaussian splatting, presented in Equation (4). Our approach adopts the Hexplane structure for deformation, similar to [61].

4 Uncertanty-Aware 4D Gaussian Splatting

4.1 Uncertainty-Aware Regularization

We now discuss the proposed uncertainty-aware regularization technique designed for the balance between reconstruction quality on training images and generalization to unseen views.

Uncertainty quantification We first estimate how informative each Gaussian primitive is for reconstruction, based on its visibility from all pixels in training images and its opacity, which is given by

where r is a pixel in a training image I ∈ T , and ω π k ( r ) is the contribution of each Gaussian primitive γ k to pixel r during α -blending, as described in Equation (5). For the computation of this value, we customize the CUDA kernel to modify the backward process of the 3DGS [23].

The parameters of an informative Gaussian primitive are typically estimated accurately with high confidence. Conversely, a Gaussian primitive that is not properly supported by training images struggles with low accuracy and high uncertainty of its parameter estimation. Based on these observations, the uncertainty of each Gaussian primitive, U k , is defined as

where the sigmoid function is used to bound and normalize C k and { c 0 , c 1 } control the inflection point shift and the slope of the sigmoid function, respectively. Given an arbitrary unseen viewpoint, a 2D uncertainty map U is constructed using α -blending as follows:

where r is a pixel in the uncertainty map U . We employ the estimated uncertainty map for the adaptive regularization to unseen views. Specifically, we adopt a diffusion prior as well as a depth smoothing prior and the details of these two priors are discussed next.

Uncertainty-aware diffusion-based regularization To render natural-looking images for novel views and times, we incorporate Stable Diffusion [44] into our pipeline. We begin by generating text prompts from the training frames using the vision-language model BLIP [28]. These prompts guide fine-tuning of the diffusion model via DreamBooth [45] with the training images, which aligns the model's understanding to the specific content in the training images as discussed in the image-to-3D reconstruction algorithm [42]. Using this fine-tuned model, we produce a refined image, I DDIM from the rendered image ˆ I for novel views or times. Specifically, we first encode ˆ I into the latent space using the latent diffusion encoder Enc, then perturb it into a noisy latent representation as follows:

where ¯ α t is a scalar value that controls the noise level and t is a diffusion time step. Similar to SDEdit [32], we generate I DDIM by performing the DDIM sampling [54] over k = glyph[floorleft] 50 · t T glyph[floorright] steps and running the diffusion decoder Dec as follows:

where v is the text embedding of the prompt from BLIP. Applying a reconstruction loss between ˆ I and I DDIM is helpful for generating natural-looking images for unseen views; however, it may compromise reconstruction quality because I DDIM sometimes contains misaligned context with the actual 3D scene inherent in the training images. To achieve accurate and realistic reconstruction by balancing the two properties, we propose uncertainty-aware diffusion loss, L UA-diff. This loss applies the estimated uncertainty to the computation of the reconstruction error between the synthesized image, ˆ I and the corresponding DDIM-sampled image, I DDIM, where uncertain regions are more regularized than low-uncertainty regions-where the training data already provide sufficient reconstruction detail-as follows:

where U is the uncertainty map of the unseen view and · denotes element-wise product. Since the DDIM sampling for generating I DDIM is time-consuming, performing it every iteration could slow down the training process, which is not desirable. To avoid the computational burden for training, we randomly sample 200 images at the beginning of every 2,000 iterations, refine them by DDIM, and store them in memory. During the next 2,000 iterations, we utilize the stored images to compute the diffusion-based regularization term, L UA-diff .

Uncertainty-aware depth smoothing regularization We introduce an additional regularization term to encourage smooth depth predictions in uncertain regions. To this end, we first generate a depth map D for unseen views using an α -blending method as follows:

where r ∈ R 2 is a pixel coordinate in the depth map ˆ D , and d k denotes the depth at the center of the k th Gaussian with respect to the camera center. We employ the total variation to regularize the estimated depth map ˆ D , which promotes smooth depth transitions between neighboring pixels. However, the uniform total variation loss over all pixels produces blur artifacts on accurately predicted regions, resulting in large reconstruction error. To address this drawback, we propose an uncertaintyaware total variation loss, L UA-TV, which applies stronger smoothing to high-uncertainty regions for noise reduction in the depth map, while leaving low-uncertainty areas unregularized to preserve details. Our uncertainty-aware total variation loss is given by

where ˆ D i,j is the estimated depth at pixel ( i, j ) and u r and u c respectively denote the sums of the average uncertainties of vertically and horizontally adjacent pixels, in other words, u r = ∑ i,j U i,j + U i +1 ,j 2 and u c = ∑ i,j U i,j + U i,j +1 2 .

4.2 Dynamic Region Densification

Existing 4D Gaussian Splattings initialize Gaussian primitives using point clouds obtained from Structure from Motion (SfM) [47]. However, since SfM assumes static scenes, it is fundamentally unable to reconstruct dynamic regions, particularly those involving rapid motion, as shown in Figure 2c. This failure occurs because the algorithm treats dynamic regions as noise, leaving these areas without initialized primitives. Such incomplete initialization disrupts the training process, causing primitives in static regions to be repeatedly cloned and split in an attempt to fill the dynamic areas. This leads to an excessive number of primitives and, in some case, out-of-memory issues.

(a) Training image

(b) Scene flow

(c) Initialization from SfM

(d) Dynamic region densification

To address this limitation, we propose a dynamic region densification that initializes additional Gaussian primitives Γ ′ = { γ ′ 1 , ..., γ ′ K } in dynamic regions. To this end, we first identify dynamic pixels in the training images using scene flows [71] and randomly select a subset of the pixels. For each selected pixel r , the corresponding Gaussian primitive γ ′ k is initialized with position µ k and feature vector f k as:

where D ( r ) is the depth value of pixel r estimated from [71], and π -1 is the inverse projection function that reprojects r into 3D space. The feature vector f k encodes the information of the k th Gaussian primitive, where its first three components are set to the RGB colors of pixel r in an image I and the rest of dimensions, which are optional for the spherical harmonics representation, are padded to zeros. This method provides reasonable placements of Gaussian primitives in dynamic regions as shown in Figure 2d.

4.3 Data-Driven Losses

Since the dynamic scene reconstruction from casually recorded monocular video is a highly ill-posed problem, we apply the additional data-driven losses based on depth and flow maps. The depth-driven loss is defined by the difference between the depth maps ˆ D and D , which are respectively obtained via the α -blending shown in Equation (13) and the algorithm proposed in [71], as shown in the following equation:

The flow-driven loss is analogously defined. Given two frames Π and Π ′ , the flow of a pixel r is estimated using α -blending, which is expressed by

where F Π → Π ′ k represents the deformation of the k th Gaussian primitive from frame Π to frame Π ′ in the projected image space, where ( π, t ) and ( π ′ , t ′ ) denote the projection function and timestamp associated with frames Π and Π ′ , respectively. Similar to L depth, the data-driven loss for flow is defined by

where F Π → Π ′ is an optical flow map obtained from RAFT [57].

The final data-driven loss, L data, is defined as the sum of the two data-driven loss terms, as shown in the following equation:

4.4 Total Loss

To train our uncertainty-award 4D Gaussian splatting models, the total loss function is given by

L = L recon + λ grid L grid + λ data L data + λ UA-diff L UA-diff ++ λ UA-TV L UA-TV , (20) where L recon is the standard reconstruction loss based on training images, L grid is a loss term associated with the Hexplane-based deformation adopted in our baseline [61], and { λ grid , λ data , λ UA-diff , λ UA-TV } are balancing hyperparameters for individual loss terms.

5 Experiments

This section compares the proposed method, referred to as UA-4DGS, with existing 4D Gaussian splatting algorithms including D-3DGS [68], Zhan et al. [31], and 4DGS [61]. Our method is implemented based on the official code of 4DGS [61] and tested on a single RTX A5000 GPU.

5.1 Settings

Dataset Our primary goal is to reconstruct dynamic scenes from casually recorded monocular videos, for which we use DyCheck [14] as our main dataset. This dataset consists of monocular videos captured with a single handheld camera, featuring scenes with fast motion to provide a challenging and realistic scenario for dynamic scene reconstruction. The DyCheck dataset includes 14 videos; however, only the half of scenesapple , block , paper-windmill , teddy , space-out , spin , and wheel -are suitable for evaluation due to the availability of held-out views.

Evaluation protocol To evaluate novel view rendering quality, we use three metrics: peak signalto-noise ratio (PSNR), structural similarity (SSIM), and a perceptual metric called learned perceptual image patch similarity (LPIPS). Additionally, since our target dataset provides a co-visibility mask, we also compute masked versions of these metrics, mPSNR, mSSIM, and mLPIPS, focusing on co-visible regions. For masked evaluations, we employ the JAX implementation provided by DyCheck [14].

5.2 Experimental Results

Table 1 presents the quantitative comparison of our algorithm against existing methods based on 4D Gaussian Splatting [68, 61, 31] and MLPs [39] on the DyCheck dataset [14]. Our approach, UA-4DGS, surpasses the performance of all other 4D Gaussian Splatting algorithms across all metrics. Figure 3 shows qualitative results on the space-out , paper-windmill , teddy , and spin scenes, where UA-4DGS synthesizes more realistic images, clearly outperforming existing 4D Gaussian Splatting algorithms.

Although Gaussian Splatting generally outperforms MLP-based approaches on multi-view or less challenging datasets, our experiments show that they fall behind MLP-based methods in our target setting based on casually recorded videos with a monocular handheld camera as presented in Table 1. This is probably because the methods based on Gaussian Splatting focus more on local optimization with respect to individual Gaussian primitives and, consequently, are prone to overfitting to training images in in-the-wild monocular scenarios.

5.3 Analysis

Generalization to static scenes To demonstrate the generality of our method in static scene reconstruction, we incorporate the proposed uncertainty-aware regularization to FSGS [72], a fewshot Gaussian Splatting algorithm for static scenes, and refer to this version of our model as UA-FSGS. We test both FSGS and UA-3DGS on the LLFF dataset [33] using three training images with five different runs. Table 2 presents quantitative comparisons where UA-3DGS outperforms existing methods, including both the original results and our reproduced ones of FSGS.

Ablation study Table 3 shows the results of the ablation study on each proposed component. Dynamic region densification improves performance compared to the data-driven loss alone, implying that better alignment of primitives with scene geometry enhances the effectiveness of the loss term. Moreover, uncertainty-aware regularization yields further improvements, where L UA-diff provides substantial benefits, and adding L UA-TV results in additional gains.

Impact of uncertainty consideration To evaluate the impact of incorporating uncertainty, we test regularization methods without uncertainty consideration, where we refer to this version as L diff and L TV, respectively. Table 4 shows both train and test performance; while L diff and L TV are still effective for test performance, they exhibit underfitting on training images with lower reconstruction performance. In contrast, by integrating uncertainty through L UA-diff and L UA-TV, we can enhance the balance between training reconstruction quality and test performance in novel view synthesis.

6 Conclusions

Weproposed a novel training framework for 4D Gaussian Splatting, targeting dynamic scenes captured from casually recorded monocular cameras. Our uncertainty-aware regularizations, which incorporate diffusion and depth-smoothness priors, effectively improve novel view synthesis performance while preserving reconstruction quality on training images. Additionally, we addressed the initialization challenges of Gaussian primitives in fast-moving scenes by introducing dynamic region densification. Our method demonstrated performance gains over baseline approaches, both in dynamic scene reconstructions and few-shot static scene reconstructions. We conducted a detailed analysis through extensive experiments, and we believe this work initiates research on an important, emerging problem in 4D Gaussian Splatting, offering valuable insights to the field.

Limitations and future work Novel view synthesis performance on casually recorded monocular videos still lags behind that on multi-view or simpler datasets, highlighting potential areas for improvement in future research. Currently, our regularization techniques rely on image-level regularization using 2D uncertainty maps; future work could enhance this by incorporating regularization in the Gaussian primitive level [64, 19] to directly leverage each Gaussian primitive's uncertainty. Additionally, our dynamic region densification does not consider temporal consistency for primitive initialization, but this issue may be addressed by integrating long-term tracking algorithms [9].

￥

(b) D-3DGS

(d) 4DGS

(e) UA-4DGS

(f) Depth (Ours)

Acknowledgments and Disclosure of Funding

This work was partly supported by Samsung Advanced Institute of Technology (SAIT), and by the Institute of Information & Communications Technology Planning & Evaluation (IITP) [No.RS2022-II220959 (No.2022-0-00959), No.RS-2021-II211343, No.RS-2021-II212068] and the National Research Foundation (NRF) [No.RS-2021-NR056445 (No.2021M3A9E408078222)] funded by the Korea government (MSIT).

References

J. An. Model development with vessl, 2023. URL https://www.vessl.ai/ . Software available from vessl.ai.

J. Bai, L. Huang, W. Gong, J. Guo, and Y. Guo. Self-nerf: A self-training pipeline for few-shot neural radiance fields. arXiv:2303.05775 , 2023.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV , 2021.

M. Bonotto, L. Sarrocco, D. Evangelista, M. Imperoli, and A. Pretto. Combinerf: A combination of regularization techniques for few-shot neural radiance field view synthesis. arXiv:2403.14412 , 2024.

A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR , 2023.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In ECCV , 2022.

J. Chung, J. Oh, and K. M. Lee. Depth-regularized optimization for 3d gaussian splatting in few-shot images. arXiv:2311.13398 , 2023.

K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR , 2022.

C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In ICCV , 2023.

Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. In ICCV , 2021.

J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia , 2022.

S. Fridovich-Keil, G. Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR , 2023.

C. Gao, A. Saraf, J. Kopf, and J.-B. Huang. Dynamic view synthesis from dynamic monocular video. In ICCV , 2021.

H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa. Monocular dynamic view synthesis: A reality check. In NeurIPS , 2022.

S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In ICCV , 2021.

S. Guo, Q. Wang, Y. Gao, R. Xie, and L. Song. Depth-guided robust and fast point cloud fusion nerf for sparse input views. In AAAI , 2024.

S. Hu, K. Zhou, K. Li, L. Yu, L. Hong, T. Hu, Z. Li, G. H. Lee, and Z. Liu. Consistentnerf: Enhancing neural radiance fields with 3d consistency for sparse view synthesis. arXiv:2305.11031 , 2023.

Y.-H. Huang, Y.-T. Sun, Z. Yang, X. Lyu, Y.-P. Cao, and X. Qi. Sc-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In CVPR , 2024.

J. Hyung, S. Hong, S. Hwang, J. Lee, J. Choo, and J.-H. Kim. Effective rank analysis and regularization for enhanced 3d gaussian splatting. arXiv preprint arXiv:2406.11672 , 2024.

A. Jain, M. Tancik, and P. Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV , 2021.

W. Jiang, B. Lei, and K. Daniilidis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information. In ECCV , 2024.

D. Kanaoka, M. Sonogashira, H. Tamukoh, and Y. Kawanishi. Manifoldnerf: View-dependent image feature supervision for few-shot neural radiance fields. arXiv:2310.13670 , 2023.

B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. In ACM ToG , 2023.

I. Kim, M. Choi, and H. J. Kim. Up-nerf: Unconstrained pose prior-free neural radiance field. In NeurIPS , 2023.

M. Kim, S. Seo, and B. Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR , 2022.

M.-S. Kwak, J. Song, and S. Kim. Geconerf: Few-shot neural radiance fields via geometric consistency. In ICML , 2023.

S. Lee, L. Chen, J. Wang, A. Liniger, S. Kumar, and F. Yu. Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields. IEEE Robotics and Automation Letters , 2022.

J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML , 2022.

T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR , 2022.

Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR , 2021.

Z. Li, Z. Chen, Z. Li, and Y. Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In CVPR , 2024.

C. Meng, Y. He, Y. Song, J. Song, J. Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR , 2022.

B. Mildenhall, P. P. Srinivasan, R. Ortiz-Cayon, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. In SIGGRAPH , 2019.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV , 2020.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG , 2022.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR , 2022.

X. Pan, Z. Lai, S. Song, and G. Huang. Activenerf: Learning where to see with uncertainty estimation. In ECCV , 2022.

K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV , 2021.

K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph. , 2021.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, et al. Pytorch: An imperative style, high-performance deep learning library, 2019.

A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR , 2021.

G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR , 2024.

B. Roessle, J. T. Barron, B. Mildenhall, P. P. Srinivasan, and M. Nießner. Dense depth priors for neural radiance fields from sparse input views. In CVPR , 2022.

R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR , 2022.

N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR , 2023.

L. Savant, D. Valsesia, and E. Magli. Modeling uncertainty for gaussian splatting. arXiv preprint arXiv:2403.18476 , 2024.

J. L. Schonberger and J.-M. Frahm. Structure-from-motion revisited. In CVPR , 2016.

S. Seo, Y. Chang, and N. Kwak. Flipnerf: Flipped reflection rays for few-shot novel view synthesis. In ICCV , 2023.

S. Seo, D. Han, Y. Chang, and N. Kwak. Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs. In CVPR , 2023.

R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In CVPR , 2023.

J. Shen, A. Ruiz, A. Agudo, and F. Moreno-Noguer. Stochastic neural radiance fields: Quantifying uncertainty in implicit 3d representations. In 3DV , 2021.

J. Shen, A. Agudo, F. Moreno-Noguer, and A. Ruiz. Conditional-flow nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV , 2022.

N. Somraj and R. Soundararajan. Vip-nerf: Visibility prior for sparse input neural radiance fields. In SIGGRAPH Conference , 2023.

J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502 , 2020.

J. Song, S. Park, H. An, S. Cho, M.-S. Kwak, S. Cho, and S. Kim. Därf: Boosting radiance fields from sparse input views with monocular depth adaptation. In NeurIPs , 2023.

N. Sünderhauf, J. Abou-Chakra, and D. Miller. Density-aware nerf ensembles: Quantifying predictive uncertainty in neural radiance fields. In ICRA , 2023.

Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV , 2020.

P. Truong, M.-J. Rakotosaona, F. Manhardt, and F. Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR , 2023.

G. Wang, Z. Chen, C. C. Loy, and Z. Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In CVPR , 2023.

L. Wang, J. Zhang, X. Liu, F. Zhao, Y. Zhang, Y. Zhang, M. Wu, J. Yu, and L. Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In CVPR , 2022.

G. Wu, T. Yi, J. Fang, L. Xie, X. Zhang, W. Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR , 2024.

J. Wynn and D. Turmukhambetov. Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In CVPR , 2023.

Y. Xiao, X. Liu, D. Zhai, K. Jiang, J. Jiang, and X. Ji. Sgcnerf: Few-shot neural rendering via sparse geometric consistency guidance. arXiv:2404.00992 , 2024.

T. Xie, Z. Zong, Y. Qiu, X. Li, Y . Feng, Y . Yang, and C. Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In CVPR , 2024.

D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. In ECCV , 2022.

D. Yan, J. Liu, F. Quan, H. Chen, and M. Fu. Active implicit object reconstruction using uncertainty-guided next-best-view optimization. IEEE Robotics and Automation Letters , 2023.

J. Yang, M. Pavone, and Y. Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In CVPR , 2023.

Z. Yang, X. Gao, W. Zhou, S. Jiao, Y. Zhang, and X. Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR , 2024.

J. S. Yoon, K. Kim, O. Gallo, H. S. Park, and J. Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In CVPR , 2020.

H. Zhan, J. Zheng, Y. Xu, I. Reid, and H. Rezatofighi. Activermap: Radiance field for active mapping and planning. arXiv:2211.12656 , 2022.

Z. Zhang, F. Cole, R. Tucker, W. T. Freeman, and T. Dekel. Consistent depth of moving objects in video. ACM TOG , 2021.

Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. 2024.

A Appendix

A.1 Per-Scene Breakdown

DyCheck dataset Supplementing Table 1 of the main paper, we show the experimental results from individual scenes in terms of mPSNR (PSNR), mSSIM (SSIM), and mLPIPS (LPIPS) in Table 5 As shown in the table, UA-4DGS achieves consistent improvement over its baselines across all metrics.

LLFF dataset Supplementing Table 2 of the main paper, we show the experimental results from individual scenes in terms of PSNR, SSIM, and LPIPS in Table 6. As shown in the table, UA-3DGS achieves consistent improvement over its baselines across all metrics.

A.2 Implementation Details

Our method is implemented based on the publicly available official code 1 of 4D Gaussian Splatting (4DGS) [61], using PyTorch [40]. Following our baseline, we utilize the Adam optimizer and set the resolution of the Hexplane grid to (64 , 64 , 64 , 150) . For grid smoothness in the Hexplane, we follow the default value of 4DGS. We use the DyCheck dataset 2 as our primary dataset, containing causally captured monocular videos. For diffusion finetuning, we manually select a single prompt from training frames that best represents the overall content of the video. We train our model for 40,000 iterations, where uncertainty-aware regularization is applied starting from iteration 20,000, as the refined images from diffusion model and uncertainty maps become more reliable at this stage. The coefficients ( c 0 , c 1 ) for the sigmoid function are set as 0.25 and 20 /L , respectively, where L is the number of training images. We set the balance weights for λ data , λ UA-diff, and λ UA-TV as 0.5, 0.2, and 0.01, respectively. To measure the quality of generated images, we compute mPSNR, mSSIM, and mLPIPS, leveraging visibility masks provided by the DyCheck.

For experiments on the LLFF dataset, our method is implemented based on the publicly available official code 3 of FSGS [72]. We set the balance weights for λ UA-diff, and λ UA-TV as 0.1 and 0.001, respectively, for optimal performance, applying the same hyperparameters across all scenes. All experiments are conducted in the Vessl environment [1].

A.3 Additional Qualitative Results

(b) D-3DGS

(d) 4DGS

(e) UA-4DGS

(f) Depth (Ours)

(b) D-3DGS

(d) 4DGS

(e) UA-4DGS

(f) Depth (Ours)

NeurIPS Paper Checklist

Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]

Does the paper discuss the limitations of the work performed by the authors? [Yes]

For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? [Yes]

Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? [Yes]

Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? [Yes]

Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? [Yes]

Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [Yes]

For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? [Yes]

Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?[Yes]

Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? [NA]

Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? [NA]

Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? [Yes]

Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? [NA]

For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? [NA]

Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? [NA]

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe research introduces UA-4DGS, a novel approach designed to reconstruct dynamic scenes from monocular videos. UA-4DGS compensates for information loss due to large motion and self-occlusion by incorporating diffusion priors with pixel-wise uncertainty scores. It proposes an uncertainty-aware diffusion-based regularization, which selectively enhances uncertain regions while maintaining certain ones, preventing inconsistencies with training images. It also identifies and addresses the initialization problem in Gaussian Splatting for dynamic scenes, where static scene-focused Structure from Motion (SfM) techniques fail to initialize dynamic regions properly. They propose dynamic region densification to enhance reconstruction performance and memory efficiency by adding extra Gaussian points to dynamic areas.\n\nSTRENGTHS:\nThe paper is presented very clearly, well structured, and it is in general easy to follow and understand.\n\nUA-4DGS's effectiveness is demonstrated using the DyCheck benchmark with complex object motions. Additionally, the research shows that integrating uncertainty considerations into other NeRF regularization techniques can also improve performance.\n\nWEAKNESSES:\n1. Writing Quality: The paper contains many typographical and grammatical errors, as well as notation misuse, e.g. L31, 37, 44, 79, and 124. I recommend a thorough review to correct these mistakes.\n2. Some assumptions and claims in the paper are questionable and require further validation. For instance, the basic assumption regarding the definition of certainty (\"we assume that Gaussians frequently visible in the training images have low uncertainty, while those seen less often due to motion or occlusion have high uncertainty, as they are reconstructed with lower accuracy\") and the claim about under-reconstruction in dynamic regions (\"under-reconstruction in dynamic regions negatively impacts the training process, resulting in an excessive number of gaussian points and slowing down inference time\") need to be substantiated with additional evidence. \n3. Experimental results: The experimental results presented in Figure 3 are disappointing. While the proposed UA-4DGS method seems to outperform the baselines, the results are still unsatisfactory, with significant noise and blurriness. Additionally, the comparison is based on only seven images, which is insufficient. I strongly suggest expanding the testing to include more cases to provide a more comprehensive evaluation of the model's performance.\n\nQUESTIONS:\n1. In L171, why use both L1 and L2 losses at the same time?\n2. Could you introduce more details about L176 - \"Thus, we cache 200 images every 2000 iterations for training efficiency.\"?\n\nLIMITATIONS:\nThe limitations of this paper are primarily found in the method's assumptions and the experimental demonstrations. Please refer to the weaknesses part for more details.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an uncertainty-aware regularization technique that uses diffusion priors to improve the reconstruction quality of underfitted areas. and a dynamic region densification technique to address the missing initialization problem on dynamic regions.\nExperiments verify the proposed techniques.\n\nSTRENGTHS:\n1. The proposed regularization method, which applies uncertainty-aware diffusion priors on unseen views, can reduce inconsistencies with training images.\n2. The proposed dynamic region densification can deal with the issue of missing initialization in dynamic regions.\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. The experimental setup is not reasonable. The goal of this paper is to reconstruct dynamic scenes from monocular videos. However, as shown in Sec. 7.2, only one part is relevant to evaluating the proposed method, while the other two involve the generalization of uncertainty-aware regularization. I think the authors should conduct more experiments on the dynamic scene datasets and move the two parts about generalization into the `Appendix`.\n2. As shown in L206, the authors claim the proposed method will compare with SC-GS. However, I cannot find such a comparison in Tables and Figures. Moreover, the proposed method should also be compared with Deformable 3D-GS[1].\n3. The `Appendix` should be placed after the `References`.\n\n[1] Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR. (2024)\n\nQUESTIONS:\nSee weaknesses.\nMore, \n- According to Table 2, for three components of the proposed method, *Dynamic Densification* plays the most important role (+1.3  mPSNR), $L_{\\mathrm{data}}$ is the second important (+0.49 mPSNR), and $L_{\\mathrm{UA-diff}}$ is the least important. (+0.26 mPSNR). I would like to know the performance of each component when used individually.\n- The proposed method seems to be able to change 4D-GS to any other GS-based method. I would like to know the performance of the proposed method when applying it on Deformable 3D-GS or SC-GS.\n\nLIMITATIONS:\nThis paper does **not** discuss the limitations and broader impacts.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-06"}

comments: SUMMARY:
This paper tackles the problem of modeling dynamic 3D scenes from monocular videos. To tackle the more challenging dynamic regions in 4D Gaussians, the authors propose to measure the uncertainty and guide those regions with diffusion priors, while keeping certain regions unchanged. In addition, the authors propose to re-initialize the dynamic regions for better performance.

Uncertainty of Gaussian points is measured by their visibility in the training set. Re-initialization of dynamic Gaussian is done by exploiting pre-trained depth and optical flow models.

STRENGTHS:
1. Complete ablation studies.
2. Improved quantitative results with respect to the baseline (4DGS).
3. Large quantitative improvements for the sparse view case.

WEAKNESSES:
Major
1. Qualitative results are still of low quality (similar to the previous works).
2. Model training depends on several pre-trained models on different tasks, which increases the complexity of the overall method.

Minor
1. Main performance improvement comes from the initialization strategy
2. Missing definition of co-visibility mask
3. Check the writing continuity of lines 190 and 191

QUESTIONS:
Where is the "camera information" in line 188, is it from COLMAP?

LIMITATIONS:
The authors checked the limitation box in the checklist. However, I cannot see the limitations section in the paper nor the appendix.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13