4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization

Authors: mijeong  kim
jongwoo  lim
bohyung  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 09:30:12

Abstract: Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality.
We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. 
To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions.
This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. 
We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. 
To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. 
Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.

Main paper: 
 introduction: 
 Dynamic novel View Synthesis (DVS) aims to reconstruct dynamic scenes from captured videos and generate photorealistic frames for an arbitrary new combination of a viewpoint and a time step. This task has emerged as a vital research area in the 3D vision community with rapid advancements in augmented reality and virtual reality. Early DVS research primarily relied on neural radiance fields [29, 69, 10, 13, 11, 38, 39, 41, 5, 12, 50] . In contrast, more recent methods [61, 18, 31] extend 3D Gaussian Splatting [23] to account for the additional time dimension in dynamic scenes, and these techniques are referred to as 4D Gaussian Splatting.Despite the recent success of 4D Gaussian Splatting models [61, 18, 31, 68] , their applicability remains largely limited to controlled and purpose-built environments. Most existing models are developed and tested with multi-view video setups [29, 41] . While there are several methods tackling monocular video settings, these setups are still controlled and fall short of in-the-wild scenarios. For instance, [38, 69] maintain multi-view characteristics, where the camera captures a broad arc around a slow-moving object. Also, HyperNeRF [39] relies on unrealistic train-test splits, with both sets sampled from the same video trajectory, which renders the task closer to video interpolation than genuine novel view synthesis. In this paper, we focus for the first time on more natural, real-world monocular videos [14] , where a single handheld camera moves around fast-moving objects.In casually recorded monocular videos, which often lack sufficient multi-view information, 4D Gaussian Splatting algorithms tend to overfit the training frames in real-world scenarios. To address overfitting, recent regularization techniques [26, 7, 58, 67, 25, 36, 20] can be applied to provide Existing models often use regularization techniques to introduce additional priors for unseen views, aiming to enhance novel view synthesis performance. However, these methods tend to over-regularize accurately reconstructed pixels, which degrades the reconstruction quality of training images. To address this issue, our uncertainty-aware regularization selectively focuses on uncertain regions in unseen views, preserving the quality of well-reconstructed pixels with low uncertainty. additional priors for unseen views. However, these regularization techniques often involve a balancing issue: while they effectively improve novel view synthesis performance during testing, they inherently sacrifice the reconstruction accuracy of training images. Since both the reconstruction accuracy and the novel view synthesis quality are equally important in our target task, the trade-off caused by the naïve application of the regularization techniques is not desirable.In this paper, we address this balancing issue with a simple yet effective solution: uncertainty-aware regularization. First, we quantify the uncertainty of each Gaussian primitive based on its contribution to rendering for training images. Then, a 2D uncertainty map is constructed for unseen views using an α-blending method. Regularization is selectively applied to uncertain regions, guided by the diffusion and depth smoothness priors, while low-uncertainty regions, where training data already provide sufficient reconstruction detail, are left unregularized, as illustrated in Figure 1 . This approach results in a better balance between training and test performance, achieving good performance.In real-world scenarios involving fast motions, especially in casually recorded videos, 4D Gaussian Splatting additionally faces considerable challenges with initialization. The algorithms based on Gaussian Splatting initialize Gaussian primitives using point clouds obtained by Structure from Motion (SfM) [47] . However, SfM struggles to reconstruct dynamic regions, particularly those with fast motion, often treating them as noise and leaving these areas without initialized primitives. Such an incomplete initialization disrupts training, causing primitives in static regions to be repeatedly cloned and split in an attempt to fill the dynamic areas. This can lead to an excessive number of primitives and, in some cases, out-of-memory issues. To address this limitation, we propose a dynamic region densification technique that initializes additional Gaussian primitives in dynamic regions.We address the challenging problem of 4D reconstruction from an in-the-wild monocular video recorded casually with a handheld camera-a scenario that has been rarely explored. The main contributions of this paper are summarized as follows:• showing improvements over baselines. Additionally, we validate the applicability of our method in few-shot static scene reconstruction.The rest of this paper is organized as follows. Section 2 reviews related work and Section 3 discusses the basic concepts of 4D Gaussian splatting, which builds upon 3D Gaussian Splatting by integrating deformation strategies. The details of our approach are described in Section 4, followed by the presentation of experimental results in Section 5. Finally, we conclude this paper in Section 6.2 Related Work 
dynamic novel view synthesis: 
 In recent years, significant advances have been made in novel view synthesis [34, 6, 15, 60, 35, 23] .Initially focused on static scenes, novel view synthesis has shifted towards dynamic scenes through the integration of motion modeling, now referred to as Dynamic novel View Synthesis (DVS). Early approaches [13, 10, 38, 5, 12, 50, 11] are largely driven by Neural Radiance Fields (NeRF). Some studies [13, 10] capture dynamics implicitly through temporal inputs or latent codes, and other approaches [38, 5, 12, 50, 11] focus on training the canonical NeRF and its deformation fields.The introduction of 3D Gaussian Splatting (3DGS) [23] has marked a paradigm shift in novel view synthesis, leading to the development of 4D Gaussian Splatting (4DGS) [68, 18, 61, 31] for DVS. These 4DGS methods deform canonical 3D Gaussian primitives over time using additional deformation networks, which can be based on MLPs, learnable control points, Hexplane [5, 61] , or polynomial functions. While these models excel at reconstructing dynamic scenes in controlled environments [38, 69, 29, 41, 39] , they face significant challenges when applied to casually recorded monocular videos, posing substantial hurdles for real-world applications.Casually recorded monocular videos typically provide limited multi-view information, as they are typically captured with a single handheld camera that only exhibits gentle motion. Consequently, reconstructing dynamic scenes from these videos is often regarded as a form of sparse reconstruction due to the lack of multi-view data.In the context of sparse reconstruction, various regularization techniques have been proposed to mitigate overfitting on limited training images [36, 25, 20, 48, 22, 4, 49, 53, 24, 63, 43, 8, 59, 55, 16, 65, 62, 2, 17, 16] . These approaches generally involve rendering images or depth maps for unseen views to provide additional priors. For instance, some methods leverage depth priors based on estimated depths for novel views [43, 8, 59, 55, 16, 72] , while others incorporate color smoothness constraints to enhance these views [25, 36] . Building on the success of diffusion models [44] , recent approach [62] starts to incorporate diffusion priors to produce more realistic images of novel views. While these methods effectively enhance novel view synthesis performance at test time, they inherently compromise the quality of training image reconstructions. Since both the reconstruction accuracy and the novel view synthesis quality are equality important in our target task, the trade-off caused by the naïve application of the regularization techniques is not desirable.Uncertainty estimation in novel view synthesis has primarily been explored with Neural Radiance Fields (NeRF) [29] . Pioneering approaches [37, 51, 52] re-parameterize MLP networks in NeRF using Bayesian models to compute the uncertainty of network predictions. Inspired by InfoNeRF [25] , which considers entropy along rays for few-shot NeRFs, some studies [70, 66, 27] quantify uncertainty using the entropy of density along rays from a novel view. Additionally, Density-aware NeRF Ensembles [56] measures uncertainty by examining the variance in RGB images produced by an ensemble of models.In contrast, uncertainty quantification in Gaussian Splatting [23] remains largely underexplored, with only a few works addressing this issue. Savant et al. [46] incorporate variational inference into the rendering pipeline, but this approach increases learnable parameters. Similarly, FisherRF [21] quantifies the uncertainty of Gaussian primitives by aggregating the diagonal of the Hessian matrix of the log-likelihood function; however, it is not straightforward to obtain a scalar value of uncertainty from the Hessian matrix for each Gaussian primitive. Our approach, on the other hand, directly quantifies the observed information of each Gaussian primitive by aggregating their contributions to the reconstruction of training images.Most existing works that utilize estimated uncertainty primarily focus on quantifying model predictions after training [46] or on active learning for next-view selection [21] . In contrast, our approach leverages estimated uncertainty for adaptive regularization during training, specifically targeting Gaussian Splatting in sparse reconstruction. 
preliminary: 4d gaussian splatting: 
 This section briefly overviews 3D Gaussian splatting (3DGS) [23] and explains the deformation modeling in 4D Gaussian Splatting (4DGS) [68, 18, 61, 31] for dynamic scenes.Gaussian primitive 3D Gaussian splatting has demonstrated real-time, state-of-the-art rendering quality on static scenes. It uses an explicit 3D scene representation consisting of a set of 3D Gaussian ellipsoids, denoted by Γ = {γ 1 , ..., γ K }. Each Gaussian primitive, γ k , is based on an unnormalized 3D Gaussian kernel, G k (x), parameterized by µ k and Σ k as follows:EQUATIONwhere µ k ∈ R 3 is the center position, Σ k ∈ R 3×3 is an anisotropic covariance matrix, and x ∈ R 3 is an arbitrary location in 3D space. The covariance matrix Σ k is valid only when positive semi-definite, which is challenging to enforce during optimization. To ensure this condition, we learn Σ k by decomposing it into two learnable components, a rotation matrix R k and a scaling matrix S k as follows:Σ k := R k S k S k R k .(2) In addition to the standard Gaussian parameters such as µ k , R k , and S k , the Gaussian primitive requires additional learnable parameters for its opacity, α k ∈ [0, 1], and feature, f k ∈ R d , which is typically represented by RGB colors or spherical harmonic (SH) coefficients. Thus, each Gaussian primitive is represented asγ k := (µ k , R k , S k , α k , f k ).Differentiable rasterization Before rendering with the Gaussian primitives Γ on an image space, each 3D Gaussian kernel, G k (x; µ k , Σ k ), is projected onto a 2D image space and forms a 2D Gaussian kernel, G π k (r; µ π k , Σ π k ), where π : R 3 → R 2 denotes a projection from the world coordinate to an image space. In the projected Gaussian representation, r ∈ R 2 indicates a pixel location in an image, and the 2D meanEQUATIONwhere J denotes the Jacobian of the affine approximation of the projective transformation, and W is the world-to-camera transform matrix. When rendering the primitives in Γ to a target camera, they are sorted by their depths with respect to the camera center. The color of a pixel r is then obtained by α-blending, which is given byEQUATIONwhere ω π k (r) represents a relative contribution of each Gaussian primitive to pixel r and c(f k , r) is the color of a pixel r measured along the view direction. If a feature vector f k is based on spherical harmonics coefficients, the color is decoded from f k using the view direction associated with pixel r; otherwise, the feature vector f k can be identical to the RGB color of the primitive. For more details, please refer to the original Gaussian Splatting paper [23] . Note that, following the α-blending procedure in 3DGS [23] , ω π k (r) is given byEQUATIONwhereα k G π k (r; µ π k , Σ π k )is the opacity of the k th projected primitive at the junction with a ray corresponding to pixel r and k-1 j=1 (•) is the transmittance at the primitive γ k on pixel r, which measure how much light penetrates the preceding primitives along the ray.To represent 4D scenes using Gaussian splatting, recent algorithms [68, 18, 61, 31] deform the 3D Gaussian primitives from their canonical states to a target state over time. The transformed position µ t , rotation R t , and scale S t at time t are given byEQUATIONwhere the deformation functions φ µ (•), φ r (•), and φ s (•) can be various forms, including MLPs, learnable control points [18] , Hexplane [5, 61] , or polynomial functions. A deformed 3D Gaussian primitive at time t is represented asγ k (t) := (µ t k , R t k , S t k , α k , f k ).The projection onto a 2D space follows the same procedure as the static 3D Gaussian splatting, presented in Equation (4). Our approach adopts the Hexplane structure for deformation, similar to [61] .4 Uncertanty-Aware 4D Gaussian Splatting 
uncertainty-aware regularization: 
 We now discuss the proposed uncertainty-aware regularization technique designed for the balance between reconstruction quality on training images and generalization to unseen views.We first estimate how informative each Gaussian primitive is for reconstruction, based on its visibility from all pixels in training images and its opacity, which is given byEQUATIONwhere r is a pixel in a training image I ∈ T , and ω π k (r) is the contribution of each Gaussian primitive γ k to pixel r during α-blending, as described in Equation (5) . For the computation of this value, we customize the CUDA kernel to modify the backward process of the 3DGS [23] .The parameters of an informative Gaussian primitive are typically estimated accurately with high confidence. Conversely, a Gaussian primitive that is not properly supported by training images struggles with low accuracy and high uncertainty of its parameter estimation. Based on these observations, the uncertainty of each Gaussian primitive, U k , is defined asEQUATIONwhere the sigmoid function is used to bound and normalize C k and {c 0 , c 1 } control the inflection point shift and the slope of the sigmoid function, respectively. Given an arbitrary unseen viewpoint, a 2D uncertainty map U is constructed using α-blending as follows:EQUATION)where r is a pixel in the uncertainty map U. We employ the estimated uncertainty map for the adaptive regularization to unseen views. Specifically, we adopt a diffusion prior as well as a depth smoothing prior and the details of these two priors are discussed next.To render natural-looking images for novel views and times, we incorporate Stable Diffusion [44] into our pipeline. We begin by generating text prompts from the training frames using the vision-language model BLIP [28] . These prompts guide fine-tuning of the diffusion model via DreamBooth [45] with the training images, which aligns the model's understanding to the specific content in the training images as discussed in the image-to-3D reconstruction algorithm [42] . Using this fine-tuned model, we produce a refined image, I DDIM from the rendered image Î for novel views or times. Specifically, we first encode Î into the latent space using the latent diffusion encoder Enc, then perturb it into a noisy latent representation as follows:EQUATIONwhere ᾱt is a scalar value that controls the noise level and t is a diffusion time step. Similar to SDEdit [32] , we generate I DDIM by performing the DDIM sampling [54] over k = 50 • t T steps and running the diffusion decoder Dec as follows:EQUATIONwhere v is the text embedding of the prompt from BLIP. Applying a reconstruction loss between Î and I DDIM is helpful for generating natural-looking images for unseen views; however, it may compromise reconstruction quality because I DDIM sometimes contains misaligned context with the actual 3D scene inherent in the training images. To achieve accurate and realistic reconstruction by balancing the two properties, we propose uncertainty-aware diffusion loss, L UA-diff . This loss applies the estimated uncertainty to the computation of the reconstruction error between the synthesized image, Î and the corresponding DDIM-sampled image, I DDIM , where uncertain regions are more regularized than low-uncertainty regions-where the training data already provide sufficient reconstruction detail-as follows:EQUATION)where U is the uncertainty map of the unseen view and • denotes element-wise product. Since the DDIM sampling for generating I DDIM is time-consuming, performing it every iteration could slow down the training process, which is not desirable. To avoid the computational burden for training, we randomly sample 200 images at the beginning of every 2,000 iterations, refine them by DDIM, and store them in memory. During the next 2,000 iterations, we utilize the stored images to compute the diffusion-based regularization term, L UA-diff .Uncertainty-aware depth smoothing regularization We introduce an additional regularization term to encourage smooth depth predictions in uncertain regions. To this end, we first generate a depth map D for unseen views using an α-blending method as follows:EQUATIONwhere r ∈ R 2 is a pixel coordinate in the depth map D, and d k denotes the depth at the center of the k th Gaussian with respect to the camera center. We employ the total variation to regularize the estimated depth map D, which promotes smooth depth transitions between neighboring pixels. However, the uniform total variation loss over all pixels produces blur artifacts on accurately predicted regions, resulting in large reconstruction error. To address this drawback, we propose an uncertaintyaware total variation loss, L UA-TV , which applies stronger smoothing to high-uncertainty regions for noise reduction in the depth map, while leaving low-uncertainty areas unregularized to preserve details. Our uncertainty-aware total variation loss is given byEQUATIONwhere Di,j is the estimated depth at pixel (i, j) and u r and u c respectively denote the sums of the average uncertainties of vertically and horizontally adjacent pixels, in other words, u r = i,jUi,j +Ui+1,j 2and u c = i,jUi,j +Ui,j+1 2 .Existing 4D Gaussian Splattings initialize Gaussian primitives using point clouds obtained from Structure from Motion (SfM) [47] . However, since SfM assumes static scenes, it is fundamentally unable to reconstruct dynamic regions, particularly those involving rapid motion, as shown in Figure 2c . This failure occurs because the algorithm treats dynamic regions as noise, leaving these areas without initialized primitives. Such incomplete initialization disrupts the training process, causing primitives in static regions to be repeatedly cloned and split in an attempt to fill the dynamic areas. This leads to an excessive number of primitives and, in some case, out-of-memory issues. is designed for static scenes, it fails to properly initialize Gaussian primitives in dynamic regions.Our dynamic region densification module initializes additional Gaussian primitives in the identified dynamic regions using scene flow and depth map.To address this limitation, we propose a dynamic region densification that initializes additional Gaussian primitives Γ = {γ 1 , ..., γ K } in dynamic regions. To this end, we first identify dynamic pixels in the training images using scene flows [71] and randomly select a subset of the pixels. For each selected pixel r, the corresponding Gaussian primitive γ k is initialized with position µ k and feature vector f k as:EQUATIONk , fEQUATIONwhere D(r) is the depth value of pixel r estimated from [71] , and π -1 is the inverse projection function that reprojects r into 3D space. The feature vector f k encodes the information of the k th Gaussian primitive, where its first three components are set to the RGB colors of pixel r in an image I and the rest of dimensions, which are optional for the spherical harmonics representation, are padded to zeros. This method provides reasonable placements of Gaussian primitives in dynamic regions as shown in Figure 2d .Since the dynamic scene reconstruction from casually recorded monocular video is a highly ill-posed problem, we apply the additional data-driven losses based on depth and flow maps. The depth-driven loss is defined by the difference between the depth maps D and D, which are respectively obtained via the α-blending shown in Equation ( 13) and the algorithm proposed in [71] , as shown in the following equation:EQUATIONThe flow-driven loss is analogously defined. Given two frames Π and Π , the flow of a pixel r is estimated using α-blending, which is expressed byEQUATIONwhere F Π→Π k represents the deformation of the k th Gaussian primitive from frame Π to frame Π in the projected image space, where (π, t) and (π , t ) denote the projection function and timestamp associated with frames Π and Π , respectively. Similar to L depth , the data-driven loss for flow is defined byEQUATIONwhere F Π→Π is an optical flow map obtained from RAFT [57] .The final data-driven loss, L data , is defined as the sum of the two data-driven loss terms, as shown in the following equation:EQUATIONTo train our uncertainty-award 4D Gaussian splatting models, the total loss function is given byEQUATIONwhere L recon is the standard reconstruction loss based on training images, L grid is a loss term associated with the Hexplane-based deformation adopted in our baseline [61] , and {λ grid , λ data , λ UA-diff , λ UA-TV } are balancing hyperparameters for individual loss terms. 
experiments: 
 This section compares the proposed method, referred to as UA-4DGS, with existing 4D Gaussian splatting algorithms including D-3DGS [68] , Zhan et al. [31] , and 4DGS [61] . Our method is implemented based on the official code of 4DGS [61] and tested on a single RTX A5000 GPU.Dataset Our primary goal is to reconstruct dynamic scenes from casually recorded monocular videos, for which we use DyCheck [14] as our main dataset. This dataset consists of monocular videos captured with a single handheld camera, featuring scenes with fast motion to provide a challenging and realistic scenario for dynamic scene reconstruction. The DyCheck dataset includes 14 videos; however, only the half of scenes-apple, block, paper-windmill, teddy, space-out, spin, and wheel-are suitable for evaluation due to the availability of held-out views.Evaluation protocol To evaluate novel view rendering quality, we use three metrics: peak signalto-noise ratio (PSNR), structural similarity (SSIM), and a perceptual metric called learned perceptual image patch similarity (LPIPS). Additionally, since our target dataset provides a co-visibility mask, we also compute masked versions of these metrics, mPSNR, mSSIM, and mLPIPS, focusing on co-visible regions. For masked evaluations, we employ the JAX implementation provided by DyCheck [14] .Table 1 presents the quantitative comparison of our algorithm against existing methods based on 4D Gaussian Splatting [68, 61, 31] and MLPs [39] on the DyCheck dataset [14] . Our approach, UA-4DGS, surpasses the performance of all other 4D Gaussian Splatting algorithms across all metrics. Figure 3 shows qualitative results on the space-out, paper-windmill, teddy, and spin scenes, where UA-4DGS synthesizes more realistic images, clearly outperforming existing 4D Gaussian Splatting algorithms.Although Gaussian Splatting generally outperforms MLP-based approaches on multi-view or less challenging datasets, our experiments show that they fall behind MLP-based methods in our target setting based on casually recorded videos with a monocular handheld camera as presented in Table 1 . This is probably because the methods based on Gaussian Splatting focus more on local optimization with respect to individual Gaussian primitives and, consequently, are prone to overfitting to training images in in-the-wild monocular scenarios.Generalization to static scenes To demonstrate the generality of our method in static scene reconstruction, we incorporate the proposed uncertainty-aware regularization to FSGS [72] , a fewshot Gaussian Splatting algorithm for static scenes, and refer to this version of our model as UA-FSGS. We test both FSGS and UA-3DGS on the LLFF dataset [33] using three training images with five different runs. Table 2 presents quantitative comparisons where UA-3DGS outperforms existing methods, including both the original results and our reproduced ones of FSGS.Ablation study Table 3 shows the results of the ablation study on each proposed component. Dynamic region densification improves performance compared to the data-driven loss alone, implying that better alignment of primitives with scene geometry enhances the effectiveness of the loss term. Moreover, uncertainty-aware regularization yields further improvements, where L UA-diff provides substantial benefits, and adding L UA-TV results in additional gains.Table 1 : Quantitative comparisons on a challenging dataset, DyCheck. Our approach shows the best performance among 4D Gaussian Splatting-based methods. However, Gaussian Splatting is generally worse than MLP-based methods in more challenging settings with casually recorded videos using a monocular handheld camera. Impact of uncertainty consideration To evaluate the impact of incorporating uncertainty, we test regularization methods without uncertainty consideration, where we refer to this version as L diff and L TV , respectively. Table 4 shows both train and test performance; while L diff and L TV are still effective for test performance, they exhibit underfitting on training images with lower reconstruction performance. In contrast, by integrating uncertainty through L UA-diff and L UA-TV , we can enhance the balance between training reconstruction quality and test performance in novel view synthesis. 
conclusions: 
 We proposed a novel training framework for 4D Gaussian Splatting, targeting dynamic scenes captured from casually recorded monocular cameras. Our uncertainty-aware regularizations, which incorporate diffusion and depth-smoothness priors, effectively improve novel view synthesis performance while preserving reconstruction quality on training images. Additionally, we addressed the initialization challenges of Gaussian primitives in fast-moving scenes by introducing dynamic region densification. Our method demonstrated performance gains over baseline approaches, both in dynamic scene reconstructions and few-shot static scene reconstructions. We conducted a detailed analysis through extensive experiments, and we believe this work initiates research on an important, emerging problem in 4D Gaussian Splatting, offering valuable insights to the field.Limitations and future work Novel view synthesis performance on casually recorded monocular videos still lags behind that on multi-view or simpler datasets, highlighting potential areas for improvement in future research. Currently, our regularization techniques rely on image-level regularization using 2D uncertainty maps; future work could enhance this by incorporating regularization in the Gaussian primitive level [64, 19] to directly leverage each Gaussian primitive's uncertainty. Additionally, our dynamic region densification does not consider temporal consistency for primitive initialization, but this issue may be addressed by integrating long-term tracking algorithms [9] .  
 
 Appendix: A.1 Per-Scene Breakdown DyCheck dataset Supplementing Table 1 of the main paper, we show the experimental results from individual scenes in terms of mPSNR (PSNR), mSSIM (SSIM), and mLPIPS (LPIPS) in Table 5 As shown in the table, UA-4DGS achieves consistent improvement over its baselines across all metrics. Our method is implemented based on the publicly available official code1 of 4D Gaussian Splatting (4DGS) [61] , using PyTorch [40] . Following our baseline, we utilize the Adam optimizer and set the resolution of the Hexplane grid to (64, 64, 64, 150) . For grid smoothness in the Hexplane, we follow the default value of 4DGS. We use the DyCheck dataset2 as our primary dataset, containing causally captured monocular videos. For diffusion finetuning, we manually select a single prompt from training frames that best represents the overall content of the video. We train our model for 40,000 iterations, where uncertainty-aware regularization is applied starting from iteration 20,000, as the refined images from diffusion model and uncertainty maps become more reliable at this stage.The coefficients (c 0 , c 1 ) for the sigmoid function are set as 0.25 and 20/L, respectively, where L is the number of training images. We set the balance weights for λ data , λ UA-diff , and λ UA-TV as 0.5, 0.2, and 0.01, respectively. To measure the quality of generated images, we compute mPSNR, mSSIM, and mLPIPS, leveraging visibility masks provided by the DyCheck.For experiments on the LLFF dataset, our method is implemented based on the publicly available official code3 of FSGS [72] . We set the balance weights for λ UA-diff , and λ UA-TV as 0.1 and 0.001, respectively, for optimal performance, applying the same hyperparameters across all scenes. All experiments are conducted in the Vessl environment [1] . NeurIPS Paper Checklist 1. Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes] 2. Does the paper discuss the limitations of the work performed by the authors? [Yes] 3. For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? [Yes] 4. Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? [Yes] 5. Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? [Yes] 6. Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? [Yes] 7. Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? [Yes] 8. For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?[Yes] 9. Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?[Yes] 10. Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? [NA] 11. Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? [NA] 12. Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? [Yes] 13. Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? [NA] 14. For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? [NA] 15. Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained? [NA] https://github.com/hustvl/4DGaussianshttps://github.com/KAIR-BAIR/dycheckhttps://github.com/VITA-Group/FSGS 


Decision: Reviewers acknowledge the large quantitative improvements (6fVK, fqxz), comprehensive ablation studies (6fVK), the well written paper (LdSG, fqxz) and the ability to integrate diffusion priors for unseen views (LdSG). 

The paper received a "weak accept", an "accept" and a "reject". The reviewers voting for "weak accept" and "reject" after the rebuttal have reservations on that there are remaining regions that are not perfectly reconstructed. However, as the authors state, the DyCheck dataset (which has been designed carefully and is accepted in the community), is substantially difficult, and the relevant comparison here is to the baselines. Looking at Figure 3 of the paper, it immediately becomes clear that the method significantly outperforms them. Thus, the AC sees no ground for the reservations and recommends accepting the paper.

Reviews: review: paper_summary: paper_summary: This paper tackles the problem of modeling dynamic 3D scenes from monocular videos. To tackle the more challenging dynamic regions in 4D Gaussians, the authors propose to measure the uncertainty and guide those regions with diffusion priors, while keeping certain regions unchanged. In addition, the authors propose to re-initialize the dynamic regions for better performance.

Uncertainty of Gaussian points is measured by their visibility in the training set. Re-initialization of dynamic Gaussian is done by exploiting pre-trained depth and optical flow models.
strength_weakness: strength_weakness: Major
1. Qualitative results are still of low quality (similar to the previous works).
2. Model training depends on several pre-trained models on different tasks, which increases the complexity of the overall method.

Minor
1. Main performance improvement comes from the initialization strategy
2. Missing definition of co-visibility mask
3. Check the writing continuity of lines 190 and 191
questions: questions: Where is the "camera information" in line 188, is it from COLMAP?
limitations: limitations: The authors checked the limitation box in the checklist. However, I cannot see the limitations section in the paper nor the appendix.
paper_summary: paper_summary: This paper proposes an uncertainty-aware regularization technique that uses diffusion priors to improve the reconstruction quality of underfitted areas. and a dynamic region densification technique to address the missing initialization problem on dynamic regions.
Experiments verify the proposed techniques.
strength_weakness: strength_weakness: 1. The experimental setup is not reasonable. The goal of this paper is to reconstruct dynamic scenes from monocular videos. However, as shown in Sec. 7.2, only one part is relevant to evaluating the proposed method, while the other two involve the generalization of uncertainty-aware regularization. I think the authors should conduct more experiments on the dynamic scene datasets and move the two parts about generalization into the `Appendix`.
2. As shown in L206, the authors claim the proposed method will compare with SC-GS. However, I cannot find such a comparison in Tables and Figures. Moreover, the proposed method should also be compared with Deformable 3D-GS[1].
3. The `Appendix` should be placed after the `References`.

[1] Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR. (2024)
questions: questions: See weaknesses.
More, 
- According to Table 2, for three components of the proposed method, *Dynamic Densification* plays the most important role (+1.3  mPSNR), $L_{\mathrm{data}}$ is the second important (+0.49 mPSNR), and $L_{\mathrm{UA-diff}}$ is the least important. (+0.26 mPSNR). I would like to know the performance of each component when used individually.
- The proposed method seems to be able to change 4D-GS to any other GS-based method. I would like to know the performance of the proposed method when applying it on Deformable 3D-GS or SC-GS.
limitations: limitations: This paper does **not** discuss the limitations and broader impacts.
paper_summary: paper_summary: The research introduces UA-4DGS, a novel approach designed to reconstruct dynamic scenes from monocular videos. UA-4DGS compensates for information loss due to large motion and self-occlusion by incorporating diffusion priors with pixel-wise uncertainty scores. It proposes an uncertainty-aware diffusion-based regularization, which selectively enhances uncertain regions while maintaining certain ones, preventing inconsistencies with training images. It also identifies and addresses the initialization problem in Gaussian Splatting for dynamic scenes, where static scene-focused Structure from Motion (SfM) techniques fail to initialize dynamic regions properly. They propose dynamic region densification to enhance reconstruction performance and memory efficiency by adding extra Gaussian points to dynamic areas.
strength_weakness: strength_weakness: 1. Writing Quality: The paper contains many typographical and grammatical errors, as well as notation misuse, e.g. L31, 37, 44, 79, and 124. I recommend a thorough review to correct these mistakes.
2. Some assumptions and claims in the paper are questionable and require further validation. For instance, the basic assumption regarding the definition of certainty ("we assume that Gaussians frequently visible in the training images have low uncertainty, while those seen less often due to motion or occlusion have high uncertainty, as they are reconstructed with lower accuracy") and the claim about under-reconstruction in dynamic regions ("under-reconstruction in dynamic regions negatively impacts the training process, resulting in an excessive number of gaussian points and slowing down inference time") need to be substantiated with additional evidence. 
3. Experimental results: The experimental results presented in Figure 3 are disappointing. While the proposed UA-4DGS method seems to outperform the baselines, the results are still unsatisfactory, with significant noise and blurriness. Additionally, the comparison is based on only seven images, which is insufficient. I strongly suggest expanding the testing to include more cases to provide a more comprehensive evaluation of the model's performance.
questions: questions: 1. In L171, why use both L1 and L2 losses at the same time?
2. Could you introduce more details about L176 - "Thus, we cache 200 images every 2000 iterations for training efficiency."?
limitations: limitations: The limitations of this paper are primarily found in the method's assumptions and the experimental demonstrations. Please refer to the weaknesses part for more details.

Comments: title: Thanks for the clarification
Thanks for the replay
Thanks for the rebuttal
Thanks for your reply
Thank you for the rebuttal
comment: We deeply appreciate the insightful reviews and discussions throughout the review period. We will revise our paper to incorporate your valuable feedback. Regarding the reviews, we appreciate that most reviewers gave positive feedback on our work. Below, we summarize the reviews, our responses, and the discussions.


In our research, we have addressed an emerging and challenging problem of 4D-Gaussian splatting (4DGS), especially in in-the-wild settings. By proposing a novel training scheme including dynamic region densification and uncertainty-aware regularization techniques, we have improved both the qualitative and quantitative performance of existing baselines, as highlighted by Reviewer 6fVK. As our paper addresses this emerging problem, we believe it offers valuable insights and directions that will make a meaningful contribution to the field. 




Upon request by reviewer LdSG, we further investigated the compatibility and generality of our method. To verify compatibility, we integrated our approach with an existing 4DGS baseline, Deformable-3DGS, and demonstrated performance improvements. To demonstrate generality, we further tested our method on an easier dataset compared to our target dataset in the main paper. The results show that our proposed training scheme performs well in both challenging and easier settings, confirming its broad applicability without losing generality.

Reviewer fqxz raised a concern regarding the lack of verification and evidence of our claims. For verification of our uncertainty modeling strategy, we utilized the AUSE (Area Under the Sparsification Error) evaluation protocol to test our modeling strategy. The AUSE results indicate that our uncertainty quantification strategy performs more effectively in sparse settings compared to a recent uncertainty quantification technique. For evidence of dynamic region densification, we first thoroughly explained the principles of densification in Gaussian splatting. We then provided supporting evidence by analyzing xy-directional gradients during training iterations and studying the correlation between inference speed and the number of Gaussians. Our findings reveal that without dynamic region densification, over-densification occurs, which negatively impacts inference speed.

Reviewer 6fVK acknowledged the contributions of our work with a positive rating, yet both reviewers 6fVK and fqxz have raised concerns regarding the qualitative results. As previously mentioned, our paper addresses an emerging and challenging problem, offering valuable insights and directions for the field. Despite the inherent difficulty of the task, our method demonstrates superior performance compared to existing approaches, including Deformable-3DGS, 4DGS, and Spacetime, as illustrated in both the main paper and the supplementary pdf file. 

We also clarified some parts needing further explanation, as suggested by all reviewers, including caching strategy, the impact of the proposed components, and limitations.

We believe we have carefully addressed all concerns raised by the reviewers and hope our explanations provide the necessary clarity.

Thank you for your time and feedback.
We sincerely appreciate your valuable feedback. We fully understand and acknowledge your comments, yet we kindly ask that you consider our final, careful clarification regarding our contribution. Your consideration of this would be greatly appreciated.

We have addressed an **emerging and challenging problem of 4D-Gaussian splatting (4DGS)**, especially in in-the-wild settings. Despite the inherent difficulty of the task, by proposing novel training schemes, we have improved both the qualitative and quantitative performance of existing baselines in these challenging settings, as highlighted by Reviewer 6fVK. Additionally, in terms of addressing the emerging and challenging problem of 4DGS, we believe **our paper offers valuable insights and directions that will make a meaningful contribution to the field**. 

Regarding the qualitative results, we suggested referring to the response to Reviewer 6fVK in our previous response. As we have not responded directly, we would now like to summarize and emphasize the key points here. 

- Our target **in-the-wild dataset is particularly challenging**, as evidenced by existing 4DGS baselines achieving PSNRs below 15 on this dataset, compared to over 25 on other datasets. This difficulty helps explain why the overall qualitative results in this in-the-wild dataset appear lower in quality compared to other commonly used datasets. 
- In the extremely challenging dataset, although some residual blurriness remains, **UA-4DGS consistently outperforms 4DGS baselines and significantly reduces blurriness**, as illustrated in both the main paper and the supplementary pdf file. 
As detailed in our response to Reviewer LdSG, our method also enhances baseline performance on the easier NeRF-DS dataset [1], where overall PSNR is higher and noisy artifacts are almost nonexistent compared to the challenging settings for both the baseline and our method. This suggests that **the blurriness primarily results from the difficulties inherent to the in-the-wild datasets, and our method effectively enhances performance on easier datasets without such issues**.
- Our training scheme is **general and can be integrated with any 4DGS framework**, potentially enhancing performance through improved Gaussian deformation strategies. We demonstrated this compatibility in our response to Reviewer LdSG. 

We appreciate your time and thoughtful feedback, and promise to revise our paper to include the discussions and responses provided during the rebuttal period. If you have any further concerns or questions, please feel free to reply to this message.

[1] NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects. CVPR 2023
We sincerely appreciate your insightful feedback and the further positive improvement in the rating of our work. We are also grateful for your acknowledgment of the strengths of our method, particularly its advanced qualitative and quantitative performance compared to existing works. Our paper addresses the emerging and challenging problem of 4D Gaussian splatting, especially in in-the-wild settings, and we believe it provides valuable insights and directions that will make a meaningful contribution to the field.


Thank you for your time and thoughtful feedback. If you have any further concerns or questions, feel free to reply to this message.
Thank you sincerely for the valuable feedback regarding the ethical issues. The DyCheck dataset is a publicly available dataset introduced in the paper [1] published at NeurIPS 2022. For further clarification, please refer to our response to Reviewer MtiW.

[1] A reality check: monocular dynamic view synthesis. NuerIPS 2022
We appreciate the feedback regarding the ethical issues. In our research, we used the DyCheck dataset, a publicly available dataset introduced in the paper [1] published at NeurIPS 2022, without any modification. Regarding the ethical issues of the DyCheck dataset itself, we confirmed through the NeurIPS checklist provided in the paper [1] that there were no issues. 

Since the release of the DyCheck dataset, more and more papers have used it, often for tasks different from our novel view synthesis task, such as scene flow estimation [2] at ICCV 2023, scene editing [3] at CVPR 2024, and surface reconstruction [4] at CVPR 2024. Both ICCV 2023 and CVPR 2024 adhered to similar data ethics guidelines [5, 6] and these papers were published without any ethical issues. Therefore, we believe there are no ethical issues in our research as well, since we used the public dataset without any modifications.

[1] A reality check: monocular dynamic view synthesis. NeurIPS 2022

[2] Semantic Attention Flow Fields for Monocular Dynamic Scene Decomposition. ICCV 2023

[3] Instruct 4D-to-4D: Editing 4D Scenes as Pseudo-3D Scenes Using 2D Diffusion. CVPR 2024

[4] MorpheuS: Neural Dynamic 360 Surface Reconstruction from Monocular RGB-D Video. CVPR 2024 

[5] ICCV 2023 ethics guideline: https://iccv2023.thecvf.com/ethics.guidelines-362600-2-25-26.php

[6] CVPR 2024 ethics guideline: https://cvpr.thecvf.com/Conferences/2024/EthicsGuidelines
Thanks for your clarifications. I recognize that the proposed method yields better results than the other methods, but it is still far from perceptually pleasant. For this reason, I can only raise my score to weak accept.
We sincerely appreciate the valuable feedback. We have carefully considered the concerns raised in W2 regarding the evidence supporting our assumptions, and we hope that our explanations have provided the necessary clarity.

Regarding the concerns about qualitative results, as also mentioned by Reviewer 6fVK, we respectfully suggest reviewing our detailed response to Reviewer 6fVK. While we fully acknowledge this concern, we would like to emphasize that our in-the-wild monocular setting dataset is particularly challenging compared to common 4DGS settings. Despite this difficulty, our method still outperforms the existing 4DGS baselines in terms of both qualitative and quantitative results. 

If you have any further questions or would like additional clarification, please do not hesitate to contact us. We would be more than happy to provide additional information or discuss any aspect of our work in greater detail. Your feedback is deeply appreciated, and we remain fully committed to addressing any concerns you may have.
Thank you for your thoughtful feedback on our work and for improving the rating. We promise to revise our paper by considering your comment. If you have any questions related to the work, please don't hesitate to leave comments.
Thank you for your feedback on our results; the quantitative results have significantly improved, but there is still room for enhancement in the qualitative results. 

Firstly, we acknowledge this, but it’s important to consider that **our target in-the-wild dataset is particularly challenging**, as evidenced by the fact that **existing 4DGS baselines struggle to achieve a PSNR above 15 in our target dataset, whereas they often exceed 25 on other datasets**. This suggests that it is quite natural for the qualitative results to appear lower in quality compared to those from common 4DGS settings. 

Secondly, although UA-4DGS contains some blurriness in the extremely challenging dataset [1] (tackled in our main paper), it **significantly outperforms the 4DGS baselines**, as shown in the attached file. Additionally, in case relatively less challenging NeRF-DS [2] dataset, we observed that our algorithm enhances the baseline with higher fidelity, without blurriness, as demonstrated in our response to Reviewer LdSG. Thus, the blurriness is primarily due to the difficulty of the target in-the-wild datasets, and our method is also capable of enhancing performance in easier datasets.

Finally, our method focuses on the training scheme, not the 4DGS network architecture. Since **the proposed training scheme is indeed general**, it can be seamlessly integrated into existing 4DGS baselines, as demonstrated in our response to Reviewer LdSG. Therefore, we believe our algorithm could achieve even greater performance when combined with future baselines that refine their architecture for better Gaussian deformation strategies. Of course, regardless of the baseline selection, we are committed to continuing our exploration of methods to enhance 4DGS in such a challenging setting.

If you have any further questions or would like additional clarification, please do not hesitate to contact us. We would be glad to provide more information or discuss any aspect of our work in greater detail. Your feedback is deeply appreciated, and we are fully committed to addressing any concerns you may have.

[1] Monocular Dynamic View Synthesis: A Reality Check. NeurIPS 2022  

[2] NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects. CVPR 2023
I thank the authors for their efforts. I recommend this paper for acceptance.
We strive to provide comprehensive answers and hope our responses have addressed your queries. Nevertheless, we acknowledge that there might be moments of ambiguity or potential misunderstandings. Please don't hesitate to seek further clarification on any aspect of our work.

We're also grateful for the points you raised about compatibility with other 4DGS baselines in Question 2. To demonstrate this, we integrated our proposed methods into the Deformable-3DGS [1]. The table below shows quantitative results, highlighting how our method enhances the performance of another 4DGS baseline.

|Method |M-PSNR| M-SSIM| M-LPIPS|
|---|-----|-----|----|
|Deformable-3DGS [1] |13.75 | 0.398 | 0.495 | 
|Deformable-3DGS [1] + Ours  | 15.31 | 0.434 | 0.418 |

Thank you for suggesting additional monocular video datasets. We tested our algorithm on the plate scene in the NeRF-DS dataset [2], as shown in the table below, further demonstrating the generality of our method. We believe our approach can also be applied to other datasets, including [3,4]. We will incorporate these insights into our revised paper.

|Method |PSNR | SSIM	| LPIPS|
|---|-----|-----|----|
|Deformable-3DGS [1] | 20.48 | 0.812     | 0.222  |
|Deformable-3DGS [1] + Ours  |20.74   |0.814 | 0.213 |

[1] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction. CVPR 2024

[2] NeRF-DS: Neural Radiance Fields for Dynamic Specular Objects. CVPR 2023

[3] Dynamic view synthesis from dynamic monocular video. ICCV 2021

[4] Unbiased 4d: Monocular 4d reconstruction with a neural deformation model. CVPR 2023
Dear Authors,

I appreciate your efforts in the rebuttal and for providing additional demonstrations. I have carefully reviewed your reply, particularly the examples in the attached PDF. While some cases look very good, most of them appear blurry and noisy in the details. I believe the method still has room for improvement.

Thank you.
Thanks for your answers. I appreciate the additional results, but, as other reviewers pointed out, the qualitative performance is below what NeurIPS expects, making me believe this work still needs more refinement.
Thank the authors for their detailed rebuttal! It helped with most of my concerns. 

However, I still think it is not enough to evaluate the proposed method on only one dataset of dynamic scenes.

There are some datasets recommended:

1. NeRF-DS dataset used by Deformable 3D-GS

2. NVIDIA Dynamic dataset used by DynamicNeRF[1]

3. Unbiased4D Dataset [2]

- [1] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021

- [2] Johnson, Erik, et al. "Unbiased 4d: Monocular 4d reconstruction with a neural deformation model." In CVPR, 2023.