4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization
Authors: Mijeong Kim
Jongwoo Lim
Bohyung Han
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic Novel View Synthesis
4D Gaussian splatting
Sparse reconstruction
in-the-wild
Primary area: machine_vision

Abstract: Novel view synthesis of dynamic scenes is becoming important in various applications, including augmented and virtual reality.
We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. 
To overcome the overfitting problem of existing work for these real-world videos, we introduce an uncertainty-aware regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness on such regions.
This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. 
We also identify the initialization problem of 4DGS in fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. 
To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. 
Our experiments show that the proposed method improves the performance of 4DGS reconstruction from a video captured by a handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.

4D Gaussian Splatting in the Wild with Uncertainty-Aware Regularization
Mijeong Kim mi jeong. kim@snu. ac kr
Jongwoo L.m jongwoo. 1imemmmmmm.mmmm ac
Bohyung Han 1.3
Chhanesnu.co kr
ECE, ME, and TIPAI, Seoul National University, South Korea
Abstract
Novel view synthesis of dynamic scenes is becoming important in various applica- tions, including augmented and virtual reality. We propose a novel 4D Gaussian Splatting (4DGS) algorithm for dynamic scenes from casually recorded monocular videos. To overcome the overfitting problem of existing work for these real-world videos, we introduce uncertainty- regularization that identifies uncertain regions with few observations and selectively imposes additional priors based on diffusion models and depth smoothness such regions. This approach improves both the performance of novel view synthesis and the quality of training image reconstruction. We also identify the initialization problem of 4DGS fast-moving dynamic regions, where the Structure from Motion (SfM) algorithm fails to provide reliable 3D landmarks. To initialize Gaussian primitives in such regions, we present a dynamic region densification method using the estimated depth maps and scene flow. Our experiments show that the proposed method improves the performance of 4DGS reconstruction from video captured by handheld monocular camera and also exhibits promising results in few-shot static scene reconstruction.
Introduction
Dynamic novel View Synthesis (DVS) aims reconstruct dynamic scenes from captured videos and generate photorealistic frames for an arbitrary new combination of viewpoint and a time step. This task has emerged as a vital research area in the 3D vision community with rapid advancements in augmented reality and virtual reality. Early DVS research primarily relied on neural radiance
fields 22 69 10
38
50). In contrast, more recent methods (6] T8 31 extend
3D Gaussian Splatting 23 to account for the additional time dimension in dynamic scenes, and these
techniques are referred to 4D Gaussian Splatting.
Despite the recent success of 4D Gaussian Splatting models 61
18
31
68 their applicability
remains largely limited controlled and purpose-buile environments. Most existing models developed and tested with multi-view video setups 22 41 While there are several methods tackling monocular video settings, these setups still controlled and fall short of in mn-the-wil scenarios. For instance, [38] 69 maintain multi-view characteristics, where the camera captures a broad are around slow-moving object. Also, HyperNeRF [39] relies unrealistic train-test splits, with both sets sampled from the same video trajectory, which renders the task closer to video interpolation than genuine novel view synthesis. In this paper, we foeus for the first time more natural, real-world monocular videos [14), where single handheld camera moves around fast-moving objects.
In casually recorded monocular videos, which often lack sufficient multi-view information, 4D Gaussian Splatting algorithms tend to overfit the training frames in real-world scenarios. To address overfitting, recent regularization techniques [26
25
36 20
be applied to provide
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Uncertain region
needing additional priors
Luwan
Diffusion
Luwtr
Depth smoothness
Gaussian primitives
Certain region: preserved
Uncertainty
Rendered image
Figure 1: Concept of uncertainty-aware regularization. Existing models often use regularization techniques introduce additional priors for unseen views, aiming to enhance novel view synthesis performance. However, these methods tend to over-regularize accurately reconstructed pixels, which degrades the reconstruction quality of training images. To address this issue, uncertainty-awe regularization selectively focuses on uncertain regions in unseen views, preserving the quality of well-reconstructed pixels with low uncertainty.
additional priors for unseen views. However, these regularization techniques often involve a balancing issue: while they effectively improve novel view synthesis performance during testing, they inherently sacrifice the reconstruction accuracy training images. Since both the reconstruction accuracy and the novel view synthesis quality equally important in target task, the trade-off caused by the naive application of the regularization techniques is not desirable.
In this paper, we address this balancing issue with simple yet effective solution: uncertainty-aves regularization. First, we quantify the uncertainty of each Gaussian primitive based on its contribution to rendering for training images. Then, 2D uncertainty map is constructed for unseen views using an a-blending method. Regularization is selectively applied to uncertain regions, guided by the diffusion and depth smoothness priors, while low-uncertainty regions, where training data already provide sufficient reconstruction detail, are left unregularized, as illustrated in Figure || This approach results in better balance between training and test performance, achieving good performance.
In real-world scenarios involving fast motions, especially in casually recorded videos, 4D Gaussian Splatting additionally faces considerable challenges with initialization. The algorithms based on Gaussian Splatting initialize Gaussian primitives using point clouds obtained by Structure from Motion (SfM) 47]. However, SfM struggles to reconstruct dynamic regions, particularly those with fast motion, often treating them as noise and leaving these areas without initialized primitives. Such an incomplete initialization disrupts training, causing primitives in static regions to be repeatedly cloned and split in an attempt to the dynamic areas. This can lead to an excessive number of primitives and, in some cases, out-of- issues. To address this limitation, we propose dynamic region densification technique that initializes additional Gaussian primitives in dynamic regions.
We address the challenging problem 4D reconstruction from an in-the-wild monocular video recorded casually with handheld camera- scenario that has been rarely explored. The main contributions of this paper are summarized follows:
We propose an uncertainty quantification technique based on contribution to training image rendering and introduce adaptive regularization techniques based the uncertainty map, which balances between novel view synthesis performance and training image reconstruction quality.
We address the issue of incomplete initialization in dynamic regions, emphasizing the importance of proper initialization in the training process of 4D Gaussian Splatting.
We demonstrate the effectiveness of our algorithm casually recorded monocular videos, showing improvements over baselines. Additionally, we validate the applicability of our method in few-shot static scene reconstruction.
The rest of this paper is organized follows. Section 2 reviews related work and Section 3 discusses the basic concepts of 4D Gaussian splatting, which builds upon 3D Gaussian Splatting by integrating

---

deformation strategies. The details of our approach are described in Section followed by the presentation experimental results in Section Finally, we conclude this paper Section 6
2 Related Work
2.1 Dynamic Novel View Synthesis
In recent years, significant advances have been made in novel view synthesis [34 65 60
Initially focused on static scenes, novel view synthesis has shifted towards dynamic scenes through the integration of motion modeling, now referred to as Dynamic novel View Synthesis (DVS). Early 5
approaches 13
are largely driven by Neural Radiance Fields (NeRF). Some 10 capture dynamics implicitly through temporal inputs or latent codes, and other focus on training the canonical NeRF and its deformation fields.
10
38
12
50
studies
[13
approaches
38 2
50
The introduction of 3D Gaussian Splatting (3DGS)
has marked a paradigm shift in novel
view synthesis, leading to the development of 4D Gaussian Splatting (4DGS) [68] 18 61 31 for DVS. These 4DGS methods deform canonical 3D Gaussian primitives over time using additional deformation networks, which can be based on MLPs, learnable control points, Hexplane [5] 61]. or polynomial functions. While these models excel at reconstructing dynamic scenes in controlled environments [38 69 29 41 39), they face significant challenges when applied to casually recorded monocular videos, posing substantial hurdles for real-world applications.
2.2 Regularization Techniques in Sparse Reconstruction
Casually recorded monocular videos typically provide limited multi-view information, as they are typically captured with single handheld camera that only exhibits gentle motion. Consequently, reconstructing dynamic scenes from these videos often regarded as a form of sparse reconstruction due to the lack of multi-view data.
In the context of sparse reconstruction, various regularization techniques have been proposed to mitigate overfitting on limited training images (36
25 20
22
49 53
24
63
4
59 555
16 65 62
17 16). These approaches generally involve rendering images or depth maps for unseen views to provide additional priors. For instance, some methods leverage depth priors based on estimated depths for novel views [43
59 55 16, (72), while others incorporate color smoothness constraints to enhance these views 25 36). Building on the success of diffusion models [44]
recent approach (62) starts to incorporate diffusion priors to produce more realistic images of novel views. While these methods effectively enhance novel view synthesis performance test time, they inherently compromise the quality of training image reconstructions. Since both the reconstruction accuracy and the novel view synthesis quality are equality important our target task, the trade-off caused by the naïve application of the regularization techniques is not desirable.
2.3 Uncertainty Quantifications in Novel View Synthesis
Uncertainty estimation in novel view synthesis has primarily been explored with Neural Radiance Fields (NeRF) (29]. Pioneering approaches [77 52 re-paammterizee MLP networks in NeRF using Bayesian models to compute the uncertainty of network predictions. Inspired by InfoNeRF 25), which considers entropy along rays for few-shot NeRFs, some studies (70 66) 27 quantify uncertainty using the entropy of density along rays from a novel view. Additionally, Density-aware NeRF Ensembles [56) measures uncertainty by examining the variance in RGB images produced by an ensemble of models.
In contrast, uncertainty quantification in Gaussian Splatting with only a few works addressing this issue. Savant et al. 466 incorporate variational inference into the rendering pipeline, but this approach increases learnable parameters. Similarly, FisherRF [21] quantifies the uncertainty of Gaussian primitives by aggregating the diagonal of the Hessian matrix of the log-likelihood function; however, it not straightforward to obtain a scalar value of uncertainty from the Hessian matrix for each Gaussian primitive. Our approach, on the other hand, directly quantifies the observed information of each Gaussian primitive by aggregating their contributions to the reconstruction of training images.
23 remains largely underexplored,
3

---

Most existing works that utilize estimated uncertainty primarily focus on quantifying model predic-
tions after training [46] or on active learning for next-view selection [2T] contrast, our approach leverages estimated uncertainty for adaptive regularization during training, specifically targeting Gaussian Splatting in sparse reconstruction.
3
Preliminary: 4D Gaussian Splatting
This section briefty overviews 3D Gaussian splatting (3DGS)
23 and explains the deformation for dynamic scenes.
modeling 4D Gaussian Splatting (4DGS) 68 18 61 31
3.1 3D Gaussian Splatting
Gaussian primitive 3D Gaussian splatting has demonstrated real-time, sfate-of-the-an art rendering
quality static scenes. It uses explicit 3D scene representation consisting of a set of 3D Gaussian ellipsoids, denoted by {",,,..., TK) }. Each Gaussian primitive, Yk. is based on an unnormalized 3D Gaussian kernel, 9 (a), parameterized by μt and Σg as follows:
G (a: Pa: Ex) = exp
- - me) (x mee)
(1)
where μ4 E RR3 is the center position, Σ E 3x3 is an anisotropic covariance matrix, and x E R is an arbitrary location in 3D space. The covariance matrix Σg is valid only when positive semi-definite, which challenging to enforce during optimization. To ensure this condition, we learn Σn by decomposing it into two learnable components, a rotation matrix Rg and scaling matrix Sp. as follows:
Σ S R
In addition to the standard Gaussian parameters such fx: Rg. and S., the Gaussian primitive requires additional learnable parameters for its opacity, og E [0, 1]. and feature, J& E IRd, which is typically represented by RGB colors or spherical harmonic (SH) coefficients. Thus, each Gaussian primitive is represented as Y == (JuR R4, Sk, Okk, fk).
Differentiable rasterization Before rendering with the Gaussian primitives I on an image space, each 3D Gaussian kernel, GA (z:; Mgg kx), is projected onto a 2D image space and forms 2D Gaussian kernel, 9 (r; f D.), where R R denotes projection from the world coordinate to an image space. In the projected Gaussian representation, E IR indicates a pixel location in an image, and the 2D mean 14 E TR.2 and covariance D E IR.2 are given by
F = T
and k JWE, WT
where J denotes the Jacobian of the affine approximation of the projective transformation, and W is the world-to-camert transform matrix. When rendering the primitives in I' target camera, they are sorted by their depths with respect to the camera center. The color of pixel r is then obtained by a-blending, which is given by
K î(r) = Σ (( k=1
(4)
where wg (r) represents a relative contribution of each Gaussian primitive to pixel r and c(fk, k, r) is the color of pixel measured along the view direction. If feature vector f's is based on spherical harmonics coefficients, the color is decoded from fs using the view direction associated with pixel r; otherwise, the feature vector fx can be identical the RGB color of the primitive. For more details, please refer to the original Gaussian Splatting paper 23 procedure in 3DGS 233, we (r) is given by
Note that, following the a-blending
1-1
w (r) = a (r: D
II (1- (.) ==1
(5)
where (r: ) is the opacity of the thh projected primitive at the junction with a ray corresponding pixel r and IT==1 (-) is the transmittance at the primitive 7 pixel r, which measure how much light penetrates the preceding primitives along the ray.

---

3.2 Deformation Modeling in 4D Gaussian Splatting
To represent 4D scenes using Gaussian splatting, recent algorithms [68 18 61 31
deform the
Gaussian primitives from their canonical states to target state over time. The transformed position μ², rotation R', and scale S' at time are given by
( R S - (mu + (ptk 1), Rs + dr (Ree U), + d. (Sk,,,)))
(6)
where the deformation functions (n)( (,(,),, and (.).) can various forms, including MI.Ps, learnable control points [18). Hexplane [5 61]. polynomial functions. A deformed 3D Gaussian
primitive at time is represented as ya (t) == ( Ri S ag. fx). The projection onto a 2D space
follows the same procedure as the static 3D Gaussian splatting, presented in Equation 4 Our
approach adopts the Hexplane structure for deformation, similar to [6].
4 Uncertanty-Aware 4D Gaussian Splatting
4.1
Uncertainty-Awan Regularization
We now discuss the proposed uncertainty-aware regularization technique designed for the balance between reconstruction quality on training images and generalization to unseen views.
Uncertainty quantification We first estimate how informative each Gaussian primitive is for reconstruction, based on its visibility from all pixels in training images and its opacity, which is given by
CK
ΣΣ w TET
where is a pixel in a training image I E T, and wI (r) is the contribution of each Gaussian primitive Th to pixel r during a-blending, as described Equation (5. For the computation of this value, we customize the CUDA kernel to modify the backward process of the 3DGS 23
The parameters of informative Gaussian primitive are typically estimated accurately with high confidence. Conversely, Gaussian primitive that is not properly supported by training images struggles with low accuracy and high uncertainty of its parameter estimation. Based on these observations, the uncertainty of each Gaussian primitive, Uk. is defined as
U = 1 Sigmoi (C2;: co. c1),
(8)
where the sigmoid function is used to bound and normalize C₃ and fco,c) (1) control the inflection point shift and the slope of the sigmoid function, respectively. Given an arbitrary unseen viewpoint, 2D uncertainty map U is constructed using a- e-blending as follows:
U(r) Σ
W (r)
where r a pixel in the uncertainty map U. We employ the estimated uncertainty map for the adaptive regularization to unseen views. Specifically, we adopt a diffusion prior as well as depth smoothing prior and the details of these two priors are discussed next.
Uncertainty-awaat diffusion--ased regularization To render natural-looking images for novel views and times, we incorporate Stable Diffusion 44] into our pipeline. We begin by generating text prompts from the training frames using the assion-language model BLIP [28). These prompts guide fine-tuning of the diffusion model via DreamBooth (5) with the training images, which aligns the model's 's understanding the specific content in the training images as discussed in the image-to-3D reconstruction algorithm 42 Using this fine-tuned model, we produce refined image, I from the rendered image i for novel views or times. Specifically, we first encode into the latent space using the latent diffusion encoder Enc, then perturb it into noisy latent representation as follows:
x, Enc() + /1 - ā.c, where ~ N) and t E [0, T)
(10)
5

---

where a is scalar value that controls the noise level and is a diffusion time step. Similar to
SDEdit [32], we generate IDDM by performing the DDIM sampling [54] over it [50- fl steps and
running the diffusion decoder Dec as follows:
IDII Dec(DDIM(Xx e)),
(11)
where o the text embedding of the prompt from BLIP. Applying a reconstruction loss between and IDDM is helpful for generating natural-looking images for unseen views; however, it may compromise reconstruction quality because IDDIM sometimes contains misaligned context with the actual 3D scene inherent in the training images. To achieve accurate and realistic reconstruction by balancing the two properties, we propose uncertainty-awanl diffusion loss, LUA-BI- This loss applies the estimated uncertainty to the computation of the reconstruction error between the synthesized image, i and the corresponding DDIM-sampled image, IDDI, where uncertain regions more regularized than low-uncertainty regions- where the training data already provide sufficient reconstruction detail-as follows:
U IDDIM]22 +
UU .Í IDDIM)
LUA-diff
(12)
where is the uncertainty map of the unseen view and denotes clement-wise product. Since the DDIM sampling for generating IDDIM is time-consu,ing, performing it every iteration could slow down the training process, which is not desirable, To avoid the computational burden for training, we randomly sample 200 images at the beginning of every 2,000 iterations, refine them by DDIM, and store them in memory. During the next 2,000 iterations, we utilize the stored images compute the iiftusioo-bbseld regularization term, LUA-dift-
Uncerfainty-awae aware depth smoothing regularization We introduce an additional regularization term to encourage smooth depth predictions in uncertain regions. To this end, we first generate a depth map D for unseen views using an a-blending method follows:
D(r)=
Σ w (r)da,
(13)
where r E 2R2 is a pixel coordinate in the depth map D, and da denotes the depth at the center of the bthh Gaussian with respect the camera center. We employ the total variation to regularize the estimated depth map D, which promotes smooth depth transitions between neighboring pixels. However, the uniform total variation loss over all pixels produces blur artifacts on accurately predicted regions, resulting large reconstruction error. To address this drawback, we propose an uncertainty- aware total variation loss, CUA-TV. which applies stronger smoothing to high-uncertaing, regions for noise reduction in the depth map, while leaving low-uncertainty areas unregularized preserve details. Our uncertainty-awar total variation loss is given by
CUA-TV
1 Ur 1.j
U + U
DD., Di41ol +
1 ue
U
+ U.j+1 2
Di,jtal, DD.
2
(14)
where D.. is the estimated depth at pixel (i.j) and u, and u. respectively denote the sums of the aver- age uncertainties of vertically and horizontally adjacent pixels, in other words, u, Σi. U. U
and ue = U.,++U
4.2 Dynamic Region Densification
Existing 4D Gaussian Splattings initialize Gaussian primitives using point clouds obtained from Structure from Motion (SfM) [47]. However, since SfM assumes static scenes, it fundamentally unable to reconstruct dynamic regions, particularly those involving rapid motion, shown in Figure 2c This failure occurs because the algorithm treats dynamic regions as noise, leaving these areas without initialized primitives. Such incomplete initialization disrupts the training process, causing primitives in static regions to be repeatedly cloned and split in an attempt to fill the dynamic areas. This leads to excessive number of primitives and, in some case, out-of- memory issues.

---

(a) Training image (b) Scene flow
(c) Initialization from SIM
(d) Dynamic region densification
Figure 2: Visualization of the dynamic region densification on the Backpack scene. Since SIM 47 is designed for static scenes, it fails to properly initialize Gaussian primitives in dynamic regions. Our dynamic region densification module initializes additional Gaussian primitives the identified dynamic regions using scene flow and depth map.
To address this limitation, propose dynamic region densification that initializes additional Gaussian primitives I" = .... in dynamic regions. To this end, we first identify dynamic pixels in the training images using scene flows [7]) and randomly select subset the pixels. For each selected pixel r, the corresponding Gaussian primitive 7 is initialized with position μ4 and feature vector fla as:
(r.
and
f
(2) I(r),
(15)
where D(r) is the depth value pixel r estimated from 7T] and π the inverse projection function that reprojects into 3DD space. The feature vector f& encodes the information of the hath Gaussian primitive, where its first three components are set to the RGB colors of pixel r in an image and the rest of dimensions, which are optional for the spherical harmonics representation, to zeros. This method provides reasonable placements of Gaussian primitives in dynamic regions as shown in Figure 2d
padded
4.3 Data-Driven Losses
Since the dynamic scene reconstruction from casually recorded monocular video a highly ill ill-posed problem, apply the additional data-driven losses based on depth and flow maps. The depth-driven loss is defined by the difference between the depth maps D and D, which respectively obtained via the a- b--eenddigg shown in Equation (13) and the algorithm proposed in [71]. as shown in the following equation:
depph D D/1-
(16)
The flow-driven loss analogously defined. Given two frames and II' the flow of pixel is estimated using a-blending, which is expressed by
pT- (r) Σ (()
where
F M-- = ='(( - T(p)):
(17)
where 1-f1-I1' represents the deformation of the ttth Gaussian primitive from frame 11 to frame 11' in the projected image space, where (x, and (x',t') denote the projection function and timestamp associated with frames II and II', respectively. Similar to Leepth. the data-driven loss for flow is defined by
n--n!
Laow
F
lt.
(18)
where F "' is an optical flow map obtained from RAFT
[57
The final data-driven loss, Laaaa is defined the of the data-driven loss terms, shown in the following equation:
Lasta Ldepth + Caow-
(19)
7

---

4.4 Total Loss
To train our uncertainty-aweer award 4D Gaussian splatting models, the total loss function is given by
L Lrecon + Ayria + Adaaa + XuA dill CuA- + +UATTTLUATTY
(20)
where Creon is the standard reconstruction loss based on training images, Lard is a loss term associated with the cexxlane-baseed deformation adopted in our baseline [61], and {Agid, Adata AUA-diff AUA-TV) balancing hyperparameters for individual loss terms.
5 Experiments
This section compares the proposed method, referred UA-4DGS, with existing 4D Gaussian splatting algorithms including D-3DGS [[88], Zhan el al. [31) and 4DGS [6]. Our method is implemented based on the official code of 4DGS [61] and tested on a single RTX A5000 GPU.
5.1 Settings
Our primary goal to reconstruct dynamic scenes from casually recorded monocular videos, for which we use DyCheck [14] as our main dataset. This dataset consists of monocular videos captured with a single handheld camera, featuring scenes with fast motion to provide a challenging and realistic scenario for dynamic scene reconstruction. The DyCheck dataset includes block, paper- windmill, teddy, space-oul, spin,
Dataset
14 videos; however, only the half of scenes and wheel-are suitable for evaluation due to the availability of held-out views.
Evaluation protocol To evaluate novel view rendering quality, we three metrics: peak signal- to-noise ratio (PSNR), structural similarity (SSIM), and perceptual metric called learned perceptual image patch similarity (LPIPS). Additionally, since our target dataset provides co-visibility mask, we also compute masked versions of these metrics, mPSNR, MSSIM, and mLPIPS, focusing on co-visible regions. For masked evaluations, we employ the JAX implementation provided by DyCheck [1]
5.2 Experimental Results
Table presents the quantitative comparison of our algorithm against existing methods based on 4D Gaussian Splatting [68 61 31 and MLPs 33 on the DyCheck dataset [14]. Our approach,
UA-4DGS, surpasses the performance all other 4D Gaussian Splatting algorithms across all metrics. Figure 3 shows qualitative results the space-out, paper-windmill, teddy, and spin scenes, where UA-4DGS synthesizes more realistic images, clearly outperforming existing 4D Gaussian Splatting algorithms.
Although Gaussian Splatting generally outperforms MLP-based approaches on multi-vies or less challenging datasets, our experiments show that they fall behind MLP- based methods in our target setting based on casually recorded videos with a monocular handheld camera as presented in Table This is probably because the methods based on Gaussian Splatting focus more on local optimization with respect to individual Gaussian primitives and, consequently, are prone to overfitting to training images in in-the-wild monocular scenarios.
5.3 Analysis
Generalization to static scenes To demonstrate the generality of our method in static scene reconstruction, we incorporate the proposed uncertainty- regularization to FSGS [72]. few- shot Gaussian Splatting algorithm for static scenes, and refer to this version of our model as UA-FSGS. We test both FSGS and UA-3DGS on the LLFF dataset [X using three training images with five different runs. Table presents quantitative comparisons where UA-3DGS outperforms existing methods, including both the original results and our reproduced ones of FSGS.
Ablation study Table 3 shows the results of the ablation study each proposed component. Dynamic region densification improves performance compared to the data-driven loss alone, implying that better alignment of primitives with scene geometry enhances the effectiveness of the loss term. Moreover, uncertainty-aware regularization yields further improvements, where Luvan air provides substantial benefits, and adding LUA-TV results in additional gains.
8

---

Table Quantitative comparisons on a challenging dataset, DyCheck. Our approach shows the best performance among 4D Gaussian Splatting-bar based methods. However, Gaussian Splatiing is generally worse than MI.P-based methods in more challenging settings with casually recorded videos using monocular handheld camera.
Representation
Method
FPS
mPSNR
PSNR
MSSIM
SSIM
mLPIPS,
LPIPS
T-NeRF
14
16.96
16.23
0.577
0.420
0.379
0.453
NSFF
16.45
15.79
0.570
0.415
0.339
0.409
ML.
Nerfies 38
16.81
16.43
0.569
0.417
0.332
0.399
HyperNeRF 39
15.46
15.20
0.551
0.399
0.396
0.464
D-3DGS 668
12.98
12.72
0.444
0.280
0.470
0.583
Zhan et al. 31
111
13.47
13.15
0.456
0.289
0.448
0.522
Gaussian Splatting
4DGS [6]
14.14
13.90
0.465
0.297
0.430
0.508
UA-4DGS (ours)
15.25
14.89
0.488
0.325
0.390
0.476
Table 2:
Few-shot novel view synthesis results with three views for static scenes, tested the LLFF (33] dataset. Our method significantly outperforms existing methods across all metrics. FSGSi and UA-3DGS were tested over five runs, with (f) indicating reproduced results. Results for other
methods are taken from 72 and 7
Representation
Method
FPS
PSNR
SSIM
LPIPS 1
MipNeRF 3
0.21
16.11
0.401
0.46
DietNeRF
20
0.14
14.94
0.370
0.496
MLP
RegNeRF 36
0.21
19.08
0.587
0.336
FreeNeRF 67
0.21
19.63
0.612
0.308
SparseNeRF [8
0.21
19.86
0,624
0.328
3DGS 23
385
17.83
0.582
0.321
Chung al. 7
17.17
0.497
0.337
Gaussian Splatting
FSGS 72
458
20.43
0.682
0,248
FSGST 72
461
19.82
0.672
0.238
UA-3DGS (Ours)
461
20.89
0.716
0.217
Impact of uncertainty consideration To evaluate the impact of incorporating uncertainty, test regularization methods without uncertainty consideration, where we refer to this version Lait and Ltv, respectively. Table shows both train and test performance; while Lair and Liv are still effective for test performance, they exhibit underfitting on training images with lower reconstruction performance. In contrast, by integrating uncertainty through LUA dif and LUA Tv, we can enhance the balance between training reconstruction quality and test performance in novel view synthesis.
6 Conclusions
We proposed a novel training framework for 4D Gaussian Splatting, targeting dynamic scenes captured from casually recorded monocular cameras. Our uncertainty- regularizations, which incorporate diffusion and depth-smoothness priors, effectively improve novel view synthesis performance while preserving reconstruction quality training images. Additionally, addressed the initialization challenges of Gaussian primitives in fast-moving scenes by introducing dynamic region densification. Our method demonstrated performance gains baseline approaches, both in dynamic scene reconstructions and few-shot static scene reconstructions. We conducted a detailed analysis through extensive experiments, and we believe this work initiates research important, emerging problem in 4D Gaussian Splatting, offering valuable insights to the field.
Limitations and future work Novel view synthesis performance casually recorded monocu- lar videos still lags behind that on multi-view or simpler datasets, highlighting potential areas for improvement in future research. Currently, our regularization techniques rely on image-level regular- ization using 2D uncertainty maps; future work could enhance this by incorporating regularization in the Gaussian primitive level 664 to directly leverage each Gaussian primitive's 's uncertainty, Additionally, our dynamic region densification does not consider temporal consistency for primitive initialization, but this issue may be addressed by integrating long-term tracking algorithms [

---

Table 3: Ablation test results of our training schemes the spin scene in the DyCheck 14 dataset. Dynamic Dens. refers to dynamic region densification.
Method
Cama
Dynamic Dens.
LuA-ait ditt
LUNTV
mPSNR
mSSIM
mLPIPS
4DGS 61
15.25
0.424
0.419
15.74
0.444
0.373
17.04
0.463
0.375
Ours
17.32
0.474
0.355
17.37
0.481
0.342
Table 4: Quantitative comparison of regularization methods with and without uncertainty estimation on the static room scene the LLFF dataset, where FSGS [72] are used as the baseline. Incorporating uncertainty into the regularization improves novel view synthesis by enhancing the balance between reconstruction quality on training images and performance on novel views.
Train (Reconstruction)
Test (Novel View Synthesis)
Method
Uncertainty
PSNR
SSIM t
LPIPS
PSNR
SSIM f
LPIPS
FSGS 72
42.38
0.981
0.029
20.60
0.822
0.184
ESGS w/ Lair
38.40
0.986
0.035
20.40
0.811
0.198
ESGS w/ LUA-dil (Ours)
41.28
0.989
0.029
20.98
0.835
0.174
FSGS w/ LTv
38.25
0.985
0.042
20.77
0.827
0.186
FSGS w/ TV (Ours)
38.26
0.985
0.038
21.08
0.831
0.184
(a) Ground Truth
(b) D-3DGS
(c) Zhan al.
(d) 4DGS
(e) UA-4DGS
Depth (Ours)
Figure 3: Qualitative results on the space-out, winer-windmill, teddy, and spin scenes in the DyCheck dataset. UA-4DGS (Ours) shows outstanding quality of rendered images compared existing
methods, including D-3DGS 68, Zhan et al. [31, and 4DGS 61
10

---

Acknowledgments and Disclosure of Funding
This work was partly supported by Samsung Advanced Institute of Technology (SAIT), and by the Institute of Information & Communications Technology Planning & Evaluation (ITP)) [No.RS- 2022-11220959
No. N0.RS-2021-12111.44 NN..NSN-021-121222220----2--2220 and the National (N0.0021M33445440077222222 funded by the
Research Foundation (NRF)
Korea government (MSIT).
References
[1] An. Model development with vessl, 2023, URL https://www. available from vessl.ai.
Software
[2] Bai, Huang. W. Gong, J. Guo, and Guo. Self-nerf: self-training pipeline for few-shot neural radiance fields. arXiv:2303.05775 2023.
[3] J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021.
[4] M. Bonotto, L. Sarrocco, D. Evangelista, M. Imperoli, and A. Pretto. Combinerf: A combination of regularization techniques for few-shot neural radiance field view synthesis. arXiv:2403. 14412, 2024.
[5] A. Cao and J. Johnson. Hexplane: A fast representation for dynamic scenes. In CVPR, 2023.
[6] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. In ECCV, 2022.
[7] Chung, J. Oh, and K. M. Lee, Depth-regullaizee optimization for 3d gaussian splatting in few-shot images. vrXiv:2311.13399 2023.
[8] K. Deng, Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised nerf: Fewer views and faster training for free. In CVPR, 2022.
[9] C. Doersch, Y. Yang, M. Vecerik, D. Gokay, A. Gupta, Y. Aytar, J. Carreira, and A. Zisserman. Tapir: Tracking any point with per-frame initialization and temporal refinement. In ICCV, 2023.
[10] Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. ICCV, 2021.
[11] J. Fang. T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia, 2022.
[12] S. Fridovich- Keil, G. Meanti, R. Warburg, B. Recht, and Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[13] C. Gao, A. Saraf, J. Kopf, and J.-B. Huang. Dynamic view synthesis from dynamic monocular video. In ICCV, 2021.
[14] H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa. Monocular dynamic view synthesis: reality check. In NeurlPS. 2022.
[15] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering 200fps. In ICCV, 2021.
[16] Guo, Q Wang, Y. Gao, R. Xie, and L.. Song. Depth-guided robust and fast point cloud fusion nerf for sparse input views. In AAAI, 2024.
[17] S. Hu, K. Zhou, K. Li, L. Yu, I.. Hong, T. Hu, 7. Li, G. H. Lee, and 7. Liu. Consistentnerf: En- hancing neural radiance fields with 3d consistency for sparse view synthesis. arXiv:2305.1100 2023.
[18) Y.H.. Huang. Y.TT. Sun, Z. Yang, X. Lyu, Y.P. Cao, and X. Qi. Se-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In CVPR, 2024.

---

[19] J. Hyung, S. Hong. S. Hwang, J. Lee, J. Choo, and J.-H. Kim. Effective rank analysis and regularization for enhanced 3d gaussian splatting. arXiv preprint 1rXiv:246.111677 2024.
[20] A. Jain, M. Tancik, and P. Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021.
[21] W. Jiang, B. Lei, and K. Daniiliiis. Fisherrf: Active view selection and uncertainty quantification for radiance fields using fisher information. In ECCV, 2024.
[22] D. Kanaoka, M. Sonogashira, H. Tamukoh, and Y. Kawanishi. Manifoldnerf: View-dependent image feature supervision for few-shot neural radiance fields. arXiv:2310. 13670, 2023.
[23] B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis, 3d gaussian splatting for real-time radiance field rendering. In ACM ToG, 2023.
[24] I. Kim, M. Choi, and H. J. Kim. Up-nerf: Unconstrained pose prior-free neural radiance field. In NeurIPS, 2023.
[25] M. Kim, Seo, and B. Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
[26] M.-S. Kwak, J. Song, and S. Kim. Geconerf: Few-shot neural radiance fields via geometric consistency. ICML, 2023.
[27] S. Lee, L. Chen, J. Wang, A. Liniger, S. Kumar, and F. Yu. Uncertainty guided policy for active robotic 3d reconstruction using neural radiance fields. IEEE Robotics and Automation Letters, 2022.
[28] J. Li, D. Li, C. Xiong, and S. Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In ICML, 2022.
[29] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR, 2022.
[30] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In CVPR, 2021.
[31] Z. Li, Z. Chen, Z. Li, and Y. Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In CVPR, 2024.
[32] C. Meng, Y. He, Y. Song, Song, Wu, J.-Y. Zhu, and S. Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
[33] B. Mildenhall, P. P. Srinivasan, R. Ortiz Dryio, N. K. Kalantari, R. Ramamoorthi, R. Ng, and A. Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. In SIGGRAPH, 2019.
[34] B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
[35] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM TOG, 2022.
[36] M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
[37] X. Pan, Z. Lai, S. Song, and G. Huang. Activenerf: Learning where to see with uncertainty estimation. In ECCV, 2022.
[38] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. ICCV, 2021.
[39] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualle and M. Seitz. Hypernerf: A higner-dimensional representation for topologically varying neural radiance fields. ACM Trans. Graph., 2021.
12

---

[40] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, al. Pytorch: An imperative style, high-performance deep learning library, 2019.
[41] A. Pumarola, Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021.
[42] G. Qian, J. Mai, A. Hamdi, J. Ren, A. Siarohin, B. Li, H.-Y. Lee, I. Skorokhodov, P. Wonka, S. Tulyakov, et al. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. In ICLR, 2024.
[43] B. Roessle, T. Barron, Mildenhall, P. P. Srinivasan, and M. Nießner. Dense depth priors for neural radiance fields from sparse input views. In CVPR, 2022.
[44] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
[45] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, and K. Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. CVPR, 2023.
[46] L. Savant, D. Valsesia, and E. Magli. Modeling uncertainty for gaussian splatting. arXiv preprint arXiv:2403.18476 2024.
[47] J. L.. Schonberger and J.-M. Frahm. Structure-from-mooe revisited. In CVPR, 2016.
[48] S. Seo, Y. Chang, and N. Kwak. Flipnerf: Flipped reflection rays for few-shot novel view synthesis. ICCV, 2023.
[49] S. Seo, D. Han, Y. Chang, and N. Kwak. Mixnerf: Modeling a ray with mixture density for novel view synthesis from sparse inputs. In CVPR, 2023.
[50] R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In CVPR, 2023.
[51] Shen, Ruiz, A. Agudo, and F. Moreno-Noguer. Stochastic neural radiance fields: Quantify- ing uncertainty in implicit 3d representations. In 3DV, 2021.
[52] Shen, A. Agudo, F. Moreno-Noguer, and A. Ruiz. Conditional-ftow nerf: Accurate 3d modelling with reliable uncertainty quantification. In ECCV, 2022.
[53] N. Somraj and R. Soundararajan. Vip-nerf: Visibility prior for sparse input neural radiance fields. In SIGGRAPH Conference, 2023.
[54] J. Song, C. Meng, and S. Ermon. Denoising diffusion implicit models. arXiv:2010.02502, 2020.
[55] J. Song, S. Park, H. An, S. Cho, M.-S. Kwak, S. Cho, and S. Kim. Därf: Boosting radiance fields from sparse input views with monocular depth adaptation. In NeurlPs, 2023.
[56] N. Sünderhauf, J. Abou-Chakra, and D. Miller. Density-aware nerf ensembles: Quantifying predictive uncertainty neural radiance fields. ICRA, 2023.
[57] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In ECCV, 2020.
[58] P. Truong, M.-J Rakotosaona. F. Manhardt, and F. Tombari. Sparf: Neural radiance fields from sparse and noisy poses. In CVPR, 2023.
[59] G. Wang, Z. Chen, C. C. Loy, and Z. Liu. Sparsenerf: Distilling depth ranking for few-shot novel view synthesis. In CVPR, 2023.
[60] I.. Wang, J. Zhang, X. Liu, F. Zhao, Y. Zhang, Y. Zhang, M. Wu, J. Yu, and L., Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In CVPR, 2022.
[61] G. Wu, T. Yi, Fang, I. Xie, X. Zhang, Wei, W. Liu, Q. Tian, and X. Wang. 4d gaussian splatting for real-time dynamic scene rendering. In CVPR, 2024.
13

---

[62] J. Wynn and D. Turmukhambetov. Diffusionerf: Regularizing neural radiance fields with denoising diffusion models. In CVPR, 2023.
[63] Y. Xiao, X. Liu, D. Zhai, K. Jiang, Jiang, and X. Ji. Sgenerf: Few-shot neural rendering via sparse geometric consistency guidance. arXiv:2404.00999, 2024.
[64] T. Xie, Z. Zong, Y. Qiu, X. Li, Y. Feng, Y. Yang. and C. Jiang. Physgaussian: Physics-integrated 3d gaussians for generative dynamics. In CVPR, 2024.
[65] D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. Sinnerf: Training neural radiance fields on complex scenes from single image. In ECCV, 2022.
[66] D. Yan, J. Liu, Quan, H. Chen, and M. Fu. Active implicit object reconstruction using uncertainty-guicnn next-best-view optimization. IEEE Robotics and Automation Letters, 2023,
[67] J. Yang, M. Pavone, and Y. Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In CVPR, 2023.
[68] Z. Yang. X. Gao, W. Zhou, S. Jiao, Y. Zhang, and X. Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR, 2024.
[69] S. Yoon, K. Kim, O. Gallo, H. S. Park, and J. Kautz. Novel view synthesis of dynamic scenes with globally coherent depths from a monocular camera. In CVPR, 2020.
[70] H. Zhan, J. Zheng, Y. Xu, I. Reid, and H. Rezatofighi. Activermap: Radiance field for active mapping and planning. arXiv:2211.12655 2022.
[71] Z. Zhang, Cole, R. Tucker, W. T. Freeman, and T. Dekel. Consistent depth of moving objects in video. ACM TOG, 2021.
[72] Z. Zhu, Z. Fan, Y. Jiang, and Z. Wang. Fsgs: Real-time few-shot view synthesis using gaussian splatting. 2024.
14

---

A Appendix
A.1 Per-Scene Breakdown
DyCheck dataset
Supplementing Table the main paper, we show the experimental results from individual scenes in terms of mPSNR (PSNR), mSSIM (SSIM), and mLPIPS (LPIPS) in Table As shown in the table, UA-4DGS achieves consistent improvement over its baselines across all metrics.
Table 5: Breakdown results on the DyCheck dataset. The value in (-) represents the non non-masked version of the corresponding evaluation measure.
Apple mSSIM
Block
Method
mPSNR 1
mLPIPS
mPSNR
mSSIM f
mLPIPS
D-3DGS 68
16.62 (15.69)
0.711 (0.342)
0.391 (0.512)
12.59 (12.78)
0.456 (0.287)
0.553 (0.639)
Zhan el al
31
15.84 (14.78)
0.653 (0.275)
0.436 (0.543)
13.74 (13.65)
0.528 (0.385)
0.482 (0.558)
4DGS 61]
16.38 (15.35)
0.691 (0.300)
0.434 (0.536)
13.89 (14.11)
0.548 (0.377)
0.487 (0.595)
UA-4DGS (Ours)
17.00 (15.66)
0.692 (0.313)
0.375 (0.510)
15.84 (15.70)
0.588 (0.423)
0.430 (0.547)
Pandr-windmill mSSIM
teddy
Method
mPSNR
mLPIPS
mPSNR
MSSIM
mLPIPS
D-3DGS 68
11.399 (11.45)
0.198 (0.)
0.519 (0.656)
12.42 (12.69)
0.499 (0.299)
0.491 (0.645)
Zhan el al. 31
13.40 (13.45)
0.208 (0.187)
0.417 (0.456)
11,54 (12.05)
0.487 (0.259)
0,519 (0.668)
4DGS 61
14.48 (14.52)
0.208 (0.188)
0.375 (0.385)
12.39 (12.64)
0.509 (0.275)
0.486 (0.636)
UA-4DGS (Ours)
14.62 (14.67)
0.207 (0.186)
0.343 (0.351)
12.89 (13.02)
0.520 (0.291)
0.476 (0.624)
Space-out
Spin mSSIM f
Method
mPSNR 1
mSSIM
mLPIPS
mPSNR 1
mLPIPS
D-3DGS 68
13.10 (14.11)
0.499 (0.431)
0.391 (0.455)
13.30 (12.00)
0.410 (0.211)
0.477 (0.681)
Zhan et al
13.94 (14.20)
0.518 (0.464)
0.352 (0.424)
14.04 (13.32)
0.413 (0.226)
0.475 (0.552)
4DGS [1]
14,40 (14.68)
0.513 (0.447)
0.365 (0.422)
15.25 (14.77)
0.424 (0.281)
0.419 (0.511)
UA-4DGS (Ours)
16.24 (16.48)
0.546 (0.481)
0.368 (0.423)
17.37 (16.92)
0.481 (0.351)
0.342 (0.422)
Wheel
Average
Method
mPSNR
mSSIM
mLPIPS
mPSNR
mSSIM
mLPIPS
D-3DGS 68
11.40 (10.31)
0.334 (0.214)
0.466 (0.492)
12,98 (12.72)
0.444 (0.280)
0.470 (0,583)
Zhan et al. 31
11.79 (10.56)
0.385 (0.231)
0.459 (0.453)
13.47 (13.15)
0.456 (0.289)
0.448 (0.522)
4DGS 61
12.21 (11.24)
0.363 (0.211)
0.445 (0.471)
14.14 (13.90)
0.465 (0.297)
0.430 (0.508)
UA-4DGS (Ours)
12.81 (11.75)
0.385 (0.233)
0.394 (0.454)
15.25 (14.89)
0.488 (0.325)
0.390 (0.476)
LLFF dataset
Supplementing Ta the main paper, we show the experimental results from individual scenes terms of PSNR, SSTM, and LPIPS in Table As shown in the table, UA-3DGS achieves consistent improvement over its baselines across all metrics.
Table 6: Breakdown results on the LLFF dataset.
Fern SSIM
Flower SSIM
Fortress SSIM
Method
PSNR
LPIPS 4
PSNR
LPIPS
PSNR
LPIPS
FSGS 722
21.80 21.04
0.719
0.216
20.20
0.620 0.705
0.257 0.222
23.23
0.711
0.18
UA-3DGS (Ours)
0.832
0.178
21.40
23.27
0.709
0.181
Hom
Leaves SSIM
Orchids SSIM
Method
PSNR
SSIM f
IPIPS
PSNR
I.PPS
PSNR
L.IPS
FSGS
19.75 21.40
0.683
0.262
16.92
0.581 0.713
0.263 0.180
16.43 16.47
0.487 0.547
3A-3DGS (Ours)
0.705
0.222
23.22
0.307
Room
Trex SSIM
Average SSIM +
Method
PSNR
SSIM 1
I.PISS
PSNR
LPIPS
PSNR
I.IPS +
FSGS 722
20.60
0.822
0.184
19.65
0.755
0.225
19.82
0.672 0.716
0.238
UA-3DGS (Ours)
21.21
0.833
0.178
19.11
0.685
0.264
20.89
0.217
15

---

A.2 Implementation Details
Our method is implemented based on the publicly available official code of 4D Gaussian Splatting (4DGS) [6]], using PyTorch [40]. Following our baseline, we utilize the Adam optimizer and set the resolution of the Hexplane grid (64, 64, 64, 150). For grid smoothness in the Hexplane, we follow the default value of 4DGS. We use the DyCheck dataset as our primary dataset, containing causally captured monocular videos. For diffusion finetuning, we manually select single prompt from training frames that best represents the overall content of the video. We train our model for 40,000 iterations, where uncertainty-awart regularization is applied starting from iteration 20,000, as the refined images from diffusion model and uncertainty maps become more reliable at this stage. The coefficients (co, C1) for the sigmoid function are set as 0.25 and 20/ L. respectively, where is the number of training images. We set the balance weights for Adata AUA-dim- and AUA-TV as 0.5, 0.2, and 0.01, respectively. To measure the quality of generated images, we compute mPSNR, mSSIM, and mLPIPS, leveraging visibility masks provided by the DyCheck.
For experiments on the LLFF dataset, our method is implemented based on the publicly available official code of FSGS 772. We set the balance weights for AUA and λuA TV as 0.1 and 0.001, respectively, for optimal performance, applying the same hyperparameters across all scenes. All experiments are conducted in the Vessl environment [1]
A.3 Additional Qualitative Results
(a) Ground Truth
(b) D-3DGS
(c) Zhan al.
(d) 4DGS
(e) UA-4DGS
Depth (Ours)
Figure 4: Qualitative comparison between UA-4DGS and other methods tested on the DyCheck dataset. Ours achieves the outstanding quality of rendered images.
16

---

(a) Ground Truth (b) D-3DGS
(6) Zhan et al.
(d) 4DGS
(c) UA-4DGS
(f) Depth (Ours)
Figure
5: Qualitative comparison between UA-4DGS and other methods tested on the DyCheck dataset. Ours achieves the outstanding quality of rendered images.
17

---

NeurIPS Paper Checklist
1. Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
2. Does the paper discuss the limitations of the work performed by the authors? [Yes]
3. For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof? [Yes]
4. Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)? [Yes]
5. Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, described in supplemental material?
[Yes]
6. Does the paper specify all the training and test details (e.g.. data splits, hyperparameters. how they were chosen, type of optimizer, etc.) necessary to understand the results? [Yes]
7. Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments? [Yes]
8. For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? [Yes]
9. Does the research conducted in the paper conform, in every respect, with the NeurlPS Code Ethics puttts::/neuus ips wttp:cc/ppcccccceec htttiiicttthuuiiieeceeaaaiaaaaie de dee
10. Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? [NA]
11. Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? [NA]
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? [Yes]
13. Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? [NA]
14. For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? [NA]
15. Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements your country or institution) were obtained? [NA]
18


Reviews:
[{"review_id": "8718hKnB4s", "reviewer": "Reviewer_6fVK", "summary": "This paper tackles the problem of modeling dynamic 3D scenes from monocular videos. To tackle the more challenging dynamic regions in 4D Gaussians, the authors propose to measure the uncertainty and guide those regions with diffusion priors, while keeping certain regions unchanged. In addition, the authors propose to re-initialize the dynamic regions for better performance.\n\nUncertainty of Gaussian points is measured by their visibility in the training set. Re-initialization of dynamic Gaussian is done by exploiting pre-trained depth and optical flow models.", "questions": "Where is the \"camera information\" in line 188, is it from COLMAP?", "limitations": "The authors checked the limitation box in the checklist. However, I cannot see the limitations section in the paper nor the appendix.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Complete ablation studies.\n2. Improved quantitative results with respect to the baseline (4DGS).\n3. Large quantitative improvements for the sparse view case.", "weaknesses": "Major\n1. Qualitative results are still of low quality (similar to the previous works).\n2. Model training depends on several pre-trained models on different tasks, which increases the complexity of the overall method.\n\nMinor\n1. Main performance improvement comes from the initialization strategy\n2. Missing definition of co-visibility mask\n3. Check the writing continuity of lines 190 and 191", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for acknowledging the strengths of our proposed method, particularly by highlighting the significantly improved performance in both the in-the-wild 4DGS task and the static few-shot task.\n\n## W1. Low visualization quality\nOne of our main contributions addresses the overlooked issue of existing 4D-Gaussian splatting in in-the-wild scenarios. Although there exist 4DGS approaches that deal with monocular video reconstructions, they are limited to tackling videos with limited motion and/or unrealistic train/test split strategies, making the task similar to video interpolation. On the contrary, we target the Dycheck dataset, which is **more realistic and challenging than datasets used in previous research**. Consequently, our qualitative results are of low quality compared to the ones shown in other papers. To demonstrate the superiority of the proposed method, we present more qualitative results together with the outputs from other algorithms, Deformable-3DGS [1], 4DGS, and Spacetime, in the attached PDF file. Note that the additional results can only be evaluated qualitatively because their ground-truths are not available.\n\nRegardless of the quantitative and qualitative improvements, we want to emphasize the value of our work also in the application of uncertainty quantification. Until now, uncertainty quantification in novel view synthesis tasks has been limited to active learning for view selection after training. However, we are the first to leverage uncertainty in the training process, demonstrating its effectiveness in monocular video and few-shot settings.\n\n- [1] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction, CVPR2024\n\n\n## W2. Complexity of the overall method\n\nOur method relies on several off-the-shelf models such as depth and flow estimators and diffusion models. However, depth and flow estimation models are used only once per example for training. Additionally, since the learnable parameters do not increase, training complexity only increases marginally. Since all the extra models are not required for inference (rendering), our inference cost is identical to the baseline algorithm.\n\n\n## W3. Impact of proposed components\n\nWe agree that the initialization strategy (dynamic densification) has more contribution than UA-Diff, but we emphasize that dynamic densification is also our contribution. Because UA-Diff always has to be applied after dynamic densification to show its effectiveness as shown in the table below, the direct comparison between the two components is fair. Both the modules are the important parts of our algorithm.\n\n|Dynamic dens     | UA-Diff                 | M-PSNR\t| M-SSIM\t| M-LPIPS |\n|----------------------|-----------------------|---------------|--------------------|------------|\n|x                      |x \t\t        |15.74\t |0.444\t               |0.373       |\n|x                      |o \t\t        |15.71\t |0.441\t               |0.412       |\n|o                        |x \t\t        |17.04\t |0.463\t               |0.375       |\n|o                        |o \t\t        |17.30\t |0.474                |0.375       |\n\n\n## W4 & Q1 Clarification\n\nThe co-visibility mask, provided by the DyCheck dataset is used for evaluation, assigning a value of 1 to pixels to invisible regions during training. For m-PSNR, m-SSIM, and m-LPIPS, areas where the co-visibility mask is 0 are excluded from the testing process.\nThe camera information in lines 190 indicates the camera pose of the training images, which we used as provided by the DyCheck dataset.\n\n## L1. Limitations\nWe apologize for the mistake regarding the checklist and will correct this by adding the limitations. Compared to datasets that use multiview cameras, the in-the-wild monocular video setting inherently presents more challenges. Thus, we agree that the results is still blurry than other easier datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0tfZpInjW7", "reviewer": "Reviewer_LdSG", "summary": "This paper proposes an uncertainty-aware regularization technique that uses diffusion priors to improve the reconstruction quality of underfitted areas. and a dynamic region densification technique to address the missing initialization problem on dynamic regions.\nExperiments verify the proposed techniques.", "questions": "See weaknesses.\nMore, \n- According to Table 2, for three components of the proposed method, *Dynamic Densification* plays the most important role (+1.3  mPSNR), $L_{\\mathrm{data}}$ is the second important (+0.49 mPSNR), and $L_{\\mathrm{UA-diff}}$ is the least important. (+0.26 mPSNR). I would like to know the performance of each component when used individually.\n- The proposed method seems to be able to change 4D-GS to any other GS-based method. I would like to know the performance of the proposed method when applying it on Deformable 3D-GS or SC-GS.", "limitations": "This paper does **not** discuss the limitations and broader impacts.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed regularization method, which applies uncertainty-aware diffusion priors on unseen views, can reduce inconsistencies with training images.\n2. The proposed dynamic region densification can deal with the issue of missing initialization in dynamic regions.\n3. The paper is well-written and easy to follow.", "weaknesses": "1. The experimental setup is not reasonable. The goal of this paper is to reconstruct dynamic scenes from monocular videos. However, as shown in Sec. 7.2, only one part is relevant to evaluating the proposed method, while the other two involve the generalization of uncertainty-aware regularization. I think the authors should conduct more experiments on the dynamic scene datasets and move the two parts about generalization into the `Appendix`.\n2. As shown in L206, the authors claim the proposed method will compare with SC-GS. However, I cannot find such a comparison in Tables and Figures. Moreover, the proposed method should also be compared with Deformable 3D-GS[1].\n3. The `Appendix` should be placed after the `References`.\n\n[1] Yang, Z., Gao, X., Zhou, W., Jiao, S., Zhang, Y., Jin, X.: Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction. In CVPR. (2024)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for complimenting our novel training schemes, uncertainty-aware regularization, and densification technique for monocular video reconstruction, especially having complex object motions. The proposed uncertainty-aware regularization is a generic method, which is effective on both dynamic scene construction and few-shot static scene reconstruction, which is related to weakness 1. We will revise our paper by reflecting on the comments.\n\n\n## W1. More experiments on dynamic scene reconstruction\nWe presented more qualitative results in the attached PDF file in comparison to other methods. The results clearly show that the proposed approach outperforms existing techniques.\n\nIn addition, to verify the generality of uncertainty-aware regularization on dynamic scene reconstruction, we evaluate UA-TV loss in a dynamic setting. The table below shows that UA-TV enhances the performance of UA-4DGS for dynamic scene reconstruction.\nThis table is an extension of Table 2 in the main paper. \n\n### Depth regularization on dynamic scene\n|UA-Diff       | UA-TV       | M-PSNR\t| M-SSIM\t| M-LPIPS        |\n|---------------|----------------|------------------|--------------------|------------------|\n|x             |x               | 17.04\t |0.463\t\t|0.375              |\n|o             |x                |17.30\t |0.474\t\t|0.375              |\n|o              |o                  |17.42\t|0.478\t\t|0.374              |\n\n\n## W2. Missing baselines \n### Additional baselines on Dycheck datasets\n\nWe are sorry about our mistake in L206. We tried to train SC-GS [1] on the Dycheck dataset but failed to optimize the model in most scenes. This is probably because SC-GS targets multi-view synchronized videos in their main paper, which are far from the setting in our mind. Note that, unlike the multi-view setting, the control points in their algorithm are difficult to initialize with only sparse information from in-the-wild monocular video.\n\nIn the main paper and the attached file for the rebuttal, we compared our algorithm with three existing techniques [2, 3, 4]. For Deformable 3D-GS [2], we tested it on the in-the-wild Dycheck dataset and visualized rendering results in the attached file. Deformable 3D-GS produces highly blurry images due to the smoothness property of MLP, showing a low mean m-PSNR value of 12.8. Additionally, it sometimes struggles to deform accurately while maintaining a canonical status.\n\n- [1] SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes, CVPR2024,  code release: 2024.3\n- [2] Deformable 3D Gaussians for High-Fidelity Monocular Dynamic Scene Reconstruction, CVPR2024, code release: 2023.9\n- [3] 4DGS: 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering, CVPR2024, code release: 2023.12\n- [4] Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis, CVPR2024, code release: 2023.12 \n\n### Limitations of existing baselines and position of our work\n- 4DGS approaches often tackle monocular video reconstructions, but their target datasets are typically unrealistic; they contain limited motion and/or are trivially split into training and testing datasets, making them similar to video interpolation.\n- We first point out that existing 4D-Gaussian splatting paradigms perform poorly on more realistic in-the-wild videos, such as the Dycheck dataset. Our paper partially addressed the existing challenges.\n\n\n## Q1. Impact of the proposed components \nWe agree that the initialization strategy (dynamic densification) has more contribution than UA-Diff, but we emphasize that dynamic densification is also our contribution. Because UA-Diff always has to be applied after dynamic densification to show its effectiveness as shown in the table below, the direct comparison between the two components is fair. Both the modules are the important parts of our algorithm.\n\n|Dynamic dens     | UA-Diff                 | M-PSNR\t| M-SSIM\t| M-LPIPS |\n|----------------------|-----------------------|---------------|--------------------|------------|\n|x                      |x \t\t        |15.74\t |0.444\t               |0.373       |\n|x                      |o \t\t        |15.71\t |0.441\t               |0.412       |\n|o                        |x \t\t        |17.04\t |0.463\t               |0.375       |\n|o                        |o \t\t        |17.30\t |0.474                |0.375       |\n\n## Q2. Plugging into other baselines\nPlugging into other baselines, such as SC-GS or Deformable-3DGS, is a good suggestion, but unfortunately, we need more time for implementation  Currently, we observe that our dynamic densification works well on those baselines. We will discuss more extended experiments during the author-discussion period.\n\n## L1. Limitations \nCompared to datasets that use multi-view cameras, the in-the-wild setting inherently presents more challenges. Thus, the results is still blurry than other easier datasets. However, our algorithm outperforms the baselines in terms of both qualitative and quantitative results.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4cncna4h4i", "reviewer": "Reviewer_fqxz", "summary": "The research introduces UA-4DGS, a novel approach designed to reconstruct dynamic scenes from monocular videos. UA-4DGS compensates for information loss due to large motion and self-occlusion by incorporating diffusion priors with pixel-wise uncertainty scores. It proposes an uncertainty-aware diffusion-based regularization, which selectively enhances uncertain regions while maintaining certain ones, preventing inconsistencies with training images. It also identifies and addresses the initialization problem in Gaussian Splatting for dynamic scenes, where static scene-focused Structure from Motion (SfM) techniques fail to initialize dynamic regions properly. They propose dynamic region densification to enhance reconstruction performance and memory efficiency by adding extra Gaussian points to dynamic areas.", "questions": "1. In L171, why use both L1 and L2 losses at the same time?\n2. Could you introduce more details about L176 - \"Thus, we cache 200 images every 2000 iterations for training efficiency.\"?", "limitations": "The limitations of this paper are primarily found in the method's assumptions and the experimental demonstrations. Please refer to the weaknesses part for more details.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper is presented very clearly, well structured, and it is in general easy to follow and understand.\n\nUA-4DGS's effectiveness is demonstrated using the DyCheck benchmark with complex object motions. Additionally, the research shows that integrating uncertainty considerations into other NeRF regularization techniques can also improve performance.", "weaknesses": "1. Writing Quality: The paper contains many typographical and grammatical errors, as well as notation misuse, e.g. L31, 37, 44, 79, and 124. I recommend a thorough review to correct these mistakes.\n2. Some assumptions and claims in the paper are questionable and require further validation. For instance, the basic assumption regarding the definition of certainty (\"we assume that Gaussians frequently visible in the training images have low uncertainty, while those seen less often due to motion or occlusion have high uncertainty, as they are reconstructed with lower accuracy\") and the claim about under-reconstruction in dynamic regions (\"under-reconstruction in dynamic regions negatively impacts the training process, resulting in an excessive number of gaussian points and slowing down inference time\") need to be substantiated with additional evidence. \n3. Experimental results: The experimental results presented in Figure 3 are disappointing. While the proposed UA-4DGS method seems to outperform the baselines, the results are still unsatisfactory, with significant noise and blurriness. Additionally, the comparison is based on only seven images, which is insufficient. I strongly suggest expanding the testing to include more cases to provide a more comprehensive evaluation of the model's performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive comments on improving the writing and analysis of the proposed method. We answered your two concerns on evidence of our assumptions, and lack of experiments as below. We promise to revise our paper by considering your comments including writing quality. \n\n## W2. Lack of evidence on the assumptions and claims\n\n### **A. Evidence on the certainty definition**\n\n- We quantify uncertainty based on the visibility in training images. To verify our uncertainty modeling strategy, we measure the estimated uncertainty using AUSE (Area Under the Sparsification Error) [1], which is a common measure for uncertainty quantification [2], where a lower value indicates a high correlation between true error and estimated uncertainty. \n\n\n- For more valid comparisons, we also compare our method with a recent algorithm, FisherRF presented at ECCV24, which suggests a Fisher information-based uncertainty quantification. Note that FisherRF addresses a completely different task, specifically active learning for view selection, whereas our work is the **first to apply uncertainty information for regularization during training**.  Despite this, we demonstrate that our definition is more valid than FisherRF in a sparse setting by showing lower AUSE values (both with MSE and MAE). \n\n|Method \t\t| AUSE-MSE\t | AUSE-MAE|\n|-----------------|----------------|---------------|\n|Random (upper bound)\t| 0.4871\t|  0.4878|\n|FisherRF [3]\t| 0.4042\t|  0.4047 |\n|Visibility (ours)\t\t| 0.3010\t|  0.3061|\n\nAUSE-MSE and AUSE-MAE adopt Mean Squared Error (MSE) and Mean Absolute Error (MAE) for the error estimations, respectively. Note that we measure the AUSE in a few-shot setting, which can result in overall higher AUSE values due to the limited data available.\n\n- [1] Estimating and exploiting the aleatoric uncertainty in surface normal estimation. ICCV 2021\n- [2] Bayes’ Rays, CVPR 2024\n- [3] FisherRF: Active View Selection and Uncertainty Quantification for Radiance Fields using Fisher Information, ECCV2024\n\n### **B. Claims on the importance of dynamic region densification**\n\n- During training, the Gaussian splatting algorithm [4] accumulates gradients of the position values (xy-direction of image space) of each Gaussian. If these gradients exceed a predefined threshold, the Gaussians move, split, or clone to address missing Gaussians in adjacent regions. In dynamic scene reconstruction, no Gaussians initially exist in dynamic regions, causing those in static regions to repeatedly clone and split, leading to a rapid increase in Gaussian numbers.\n\n\n- To verify our claims about over-cloning and over-splitting, we measure the percentage of Gaussians whose gradients exceed the densification threshold, as shown in the table below. Without our dynamic region densification, we observed high gradients in the xy-directions, leading to over-cloning and over-splitting. For better visualization, please see Figure I in the attached PDF, where we illustrate the ratio of Gaussians exceeding the threshold with varying thresholds.\n\n|      Iteration            | w/o dynamic dens.   | w/ dynamic dens. |\n|----------------------|----------------------|--------------------|\n| 600\t\t| \t12.71%\t\t| \t3.38%          |\n| 800\t\t| \t10.09%\t\t| \t3.34%          |\n| 1200\t\t| \t11.65%\t\t| \t3.55%          |\n\t\nAlso, the below table shows the correlation between gaussian numbers and inference speed. It implies that over-densification (over-cloning & over-splitting) reduces inference speed.\n|Gaussian numbers\t| Inference speed (FPS) |\n|---------------------|--------------|\n|589102 \t\t\t| 117.36\t|\n|1132420\t\t| 81.89 |\n\n\n[4] 3D Gaussian Splatting for Real-Time Radiance Field Rendering, SIGGRAPH 2023\n\n## W3. Experimental results\n\n\nWe tackle in-the-wild settings where a monocular video contains complex motion. 4DGS approaches deal with monocular video reconstructions, but their target datasets [5,6] are too limited and unrealistic; they only contain small motion or have unrealistic train/test split strategies, making them similar to video interpolation. (The images for training and testing are sampled from a single video stream with a significantly overlapped time interval.)\n\nAgain, our target domain is in-the-wild monocular video for real-world scenarios, and it is difficult to find a dataset that satisfies the required conditions. The Dycheck dataset is an example of a realistic video dataset. Please refer to the attached file for more qualitative results.\n\n- [5] HyperNeRF: A Higher-Dimensional Representation for Topologically Varying Neural Radiance Fields. SIGGRAPH 2021 \n- [6] D-NeRF: Neural Radiance Fields for Dynamic Scenes. CVPR 2021 \n\n## Q1. L1 and L2 losses\nL1 and L2 losses are often used together in machine learning and optimization tasks to leverage the benefits of both types of loss functions and achieve better regularization. The joint use of two loss terms is helpful for the robustness to outliers (thanks to L1) and the smoothness (thanks to L2). In our problem, the synthesized images should satisfy the properties so the choice of such a loss function is reasonable.\n\n## Q2 Details of caching methods\nWhen refining rendered images with diffusion, the DDIM process is quite time-consuming if conducted every iteration. To avoid this, we sample 200 images at the beginning of every 2000 iterations, refine them with DDIM, and cache them. During the 2000 iterations, we utilize these cached images for the UA-Diffusion loss.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely appreciate your valuable comments on improving our work. Before answering your questions and concerns, we would like to highlight our contributions.

We present a novel view synthesis algorithm for dynamic scenes captured by a monocular camera. Our target task is fairly new, and we introduce a few interesting ideas such as dynamic densification and visibility-based uncertainty modeling to improve the quality of outputs. Reviewers generally acknowledged our contributions, but expressed concerns about our experiment and presentation. This rebuttal answers the critical questions and concerns raised by the reviewers. We have attached a *one-page PDF* with the characteristics of our model and qualitative results. Please take a look at it and let us know if there are additional concerns or questions.

Meta-review: Reviewers acknowledge the large quantitative improvements (6fVK, fqxz), comprehensive ablation studies (6fVK), the well written paper (LdSG, fqxz) and the ability to integrate diffusion priors for unseen views (LdSG). 

The paper received a "weak accept", an "accept" and a "reject". The reviewers voting for "weak accept" and "reject" after the rebuttal have reservations on that there are remaining regions that are not perfectly reconstructed. However, as the authors state, the DyCheck dataset (which has been designed carefully and is accepted in the community), is substantially difficult, and the relevant comparison here is to the baselines. Looking at Figure 3 of the paper, it immediately becomes clear that the method significantly outperforms them. Thus, the AC sees no ground for the reservations and recommends accepting the paper.
Decision: Accept (poster)