Fully Explicit Dynamic Gaussian Splatting
Authors: Junoh Lee
Changyeon Won
Hyunjun Jung
Inhwan Bae
Hae-Gon Jeon
Venue: NeurIPS 2024 poster (2024)
Keywords: Gaussian Splatting
Neural Rendering
Computer Vision
Primary area: machine_vision
TL;DR: 4D Gaussian Splatting with static & dynamic separation using an incrementally extensible, keyframe-based model

Abstract: 3D Gaussian Splatting has shown fast and high-quality rendering results in static scenes by leveraging dense 3D prior and explicit representations. Unfortunately, the benefits of the prior and representation do not involve novel view synthesis for dynamic motions. Ironically, this is because the main barrier is the reliance on them, which requires increasing training and rendering times to account for dynamic motions. 
In this paper, we design Explicit 4D Gaussian Splatting (Ex4DGS).
Our key idea is to firstly separate static and dynamic Gaussians during training, and to explicitly sample positions and rotations of the dynamic Gaussians at sparse timestamps. The sampled positions and rotations are then interpolated to represent both spatially and temporally continuous motions of objects in dynamic scenes as well as reducing computational cost. 
Additionally, we introduce a progressive training scheme and a point-backtracking technique that improves Ex4DGS's convergence. We initially train Ex4DGS using short timestamps and progressively extend timestamps, which makes it work well with a few point clouds. The point-backtracking is used to quantify the cumulative error of each Gaussian over time, enabling the detection and removal of erroneous Gaussians in dynamic scenes. Comprehensive experiments on various scenes demonstrate the state-of-the-art rendering quality from our method, achieving fast rendering of 62 fps on a single 2080Ti GPU.

Fully Explicit Dynamic Gaussian Splatting
Junoh Lee', Changyeon Won', Hyunjun Jung', Inhwan Bae², Hae-Gon Jeon "School of Electrical Engineering and Computer Science ?Al Graduate School Gwangju Institute of Science and Technology { juno, cywon1997 h inhwanbae Jggm gist.ac. ac.kr, kr, baegonj@gist .ww.sttac....
Abstract
3D Gaussian Splatting has shown fast and high-quality rendering results in static scenes by leveraging dense 3D prior and explicit representations. Unfortunately, the benefits of the prior and representation do not involve novel view synthesis for dynamic motions. Ironically, this is because the main barrier is the reliance on them, which requires increasing training and rendering times to account for dynamic motions. In this paper, we design Explicit 4D Gaussian Splatting (Ex4DGS), Our key idea is to firstly separate static and dynamic Gaussians during training, and to explicitly sample positions and rotations of the dynamic Gaussians at sparse timestamps. The sampled positions and rotations are then interpolated to represent both spatially and temporally continuous motions of objects in dynamic scenes as well as reducing computational cost. Additionally, we introduce a progressive training scheme and bactttbacktracking technique that improves Ex4DGS's convergence. We initially train Ex4DGS using short timestamps and progressively extend timestamps, which makes it work well with few point clouds. The point- backtracking is used to quantify the cumulative error of each Gaussian over time, enabling the detection and removal of erroneous Gaussians in dynamic scenes. Comprehensive experiments on various scenes demonstrate the state-of-the-ar rendering quality from method, achieving fast rendering of 62 fps single 2080Ti GPU.
1
Introduction
The recent flood video content has encouraged view synthesis techniques for highly engaging, visually rich content creation to maintain viewer interest. However, even short form videos require huge time for both data pre-processing, computing frame-wise 3D point clouds, and training time in novel view synthesis for dynamic motions. Furthermore, these techniques must be considered for running on mobile devices which have limited computing power and storage space. In this aspect, is crucial to not only achieve photorealistic rendering results, but also reduce the computational cost related to storage, memory and rendering pipeline. To achieve this, the spatio-temporal representation should be explicit and efficient to handle the complexity of dynamic motions in videos.
Recent methods for dynamic view synthesis are typically based on Neural Radiance Fields (NeRF) [1]. which uses implicit multi-layer perceptron (MIP) with a combination of 5-DoF spatial coordinates and the additional time axis [2, 3, 4, 5, 6, 7, 8]. The MI.P-based methods have shown high-fidelity rendering quality. However, the inevitable cost of decoding the implicit representation makes rendering critically slow. Various methods try to reduce the cost by adapting explicit representation, such as voxel [9, 10] and matrix decomposition [11, 12]. Nevertheless, since NeRF-based approaches require per-pixel ray sampling and dense sampling for each ray, it is hard to achieve real-time and high- resolution renderings.
Meanwhile, a promising alternative, 3D Gaussian Splatting (3DGS) [13], has emerged, which achieves photo-realistic rendering results with significantly faster training and rendering speeds.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Unlike NeRF-based approaches, 3DGS exploits fully explicit point-based primitives of 3D and employs a rasterization-base rendering pipeline. Recent advances [14, 15] attempt to extend 3DGS to 4D domain, handling the motion over time by storing the additional transformation information of 3D coordinates or 4D bases. However, these methods can only be trained under a restricted condition with dense point clouds. Moreover, these approaches borrow implicit representations for implementation. which lose the inherent advantage of 3DGS and make real-world applications more difficult. To become more scalable model for real-world situations, it is important to be trained under more in-the-wild conditions (i.e., sparse point cloud) with more concise representation.
In this paper, we present Explicit 4D Gaussian Splatting (Ex4DGS), keyframe interpolation- 4D Gaussian splatting method that works well in fully explicit on-time domain. Our key idea is to apply interpolation techniques under temporal explicit representation to realize the scalable 3DGS model. Temporal interpolation a widely used technique in computer graphies [16] that only stores keyframe information in video and determines smooth transitions for the rest of the frames. We select keyframes with sparse time intervals and save the additional temporal information at each keyframe which includes each Gaussian's 's position, rotation and opacity. This information is fully explicit; it is stored without any encoding process, and continuous motion is calculated to have smooth temporal transitions between adjacent keyframes. Here, we use polynomial basis interpolator for position, spherical interpolator for rotation, and a simplified Gaussian mixture model for opacity. Specifically, the polynomial basis of cubic Hermite spline (CHip) [17] is used to effectively avoid overfitting or over-smoothing problems by spanning low-degree polynomials. For rotation, we introduce Spherical Linear Interpolation (Slerp) [18] to do linear transition angles. Lastly, introduce simplified Gaussian mixture model, which allows temporal opacity to handle the appearance and disappearance of objects.
For further optimization, we reduce the computational cost by isolating dynamic points from static points, and only storing additional temporal information of dynamic points. Here, we aim for this separation to be possible without any additional inputs such as object masks [19]. To tackle this, we introduce motion-based based triggers to distinguish static and dynamic points in a scene. We first initialize all Gaussian points in a scene to be static which are assumed to move linearly. During training, static points with large movements are automatically classified as dynamic points. Next, we adopt a progressive training scheme to train our model even under sparse point cloud conditions. The progressive training, starting with short duration and scaling up, can prevent falling into local minima by reducing the sudden appearance of objects. Lastly, an additional point backtracking technique is introduced to enhance rendering quality. Detecting redundant points in a dynamic scene is challenging because we need to consider all visible timestamps. To measure accumulated errors over time, we apply the boint-backtracking technique that can track and prune high-error Gaussians in dynamic scenes.
The effectiveness of our method is validated through experiments two major real-world video datasets: Neural 3D Video dataset [7] and Technicolor dataset [20]. Experimental results demonstrate that approach significantly improves rendering results even with sparse 3D point clouds. Fur- thermore, benefiting from the proposed optimization scheme, our model only requires low storage and memory size without any auxiliary modules or tricks encode lengthy temporal information. Finally, our model achieves 62 fps on a single 2080Ti GPU on 300-frame scenes with a 1352 x 1014 resolution.
2
Related Works
2.1 Novel View Synthesis
Photorealistic novel view synthesis can be achieved using a set of densely sampled images through image-based rendering 22]. While the dense sampling of views is limited by memory constraints and results in aliased rendering, novel view synthesis has advanced with the development of neural networks. The ability of neural networks to process implicit information in images enables novel view synthesis with a sparse set of observations. Prior works on constructing multi-plane images (MPI) 24, 25] use aggregated pixel information from correspondences in sparsely sampled views. MPI representation may fail with wide angles between the camera and depth planes. This can be mitigated using geometric proxies like depth information [26, 27], plane-sweep volumes [28, 29], and meshes [30, 31J. These methods risk unrealistic view synthesis with inaccurate proxy geometry.

---

Joint optimization of proxy geometry [32, 33, 34, 35] can help, but direct mesh optimization often gets stuck in local minima due to poor loss landscape conditioning.
In recent years, continuous 3D representation through neural networks has received widespread attention. NeRF [1] is the foundational work that started this trend. It implicitly learns the shape of an object as a density field, which makes optimization via gradient descent more tractable. The robustness of such geometric neural representations enables to reconstruct accurate geometry from a few given images [36. 37, 38, 391, large-scale geometry [40, 41, 42, 43, 44, 45], disentangled textures [46, 47, 48], and material properties [49, 50, 51, 52, 53]. However, one major issue on these neural representations is also slow rendering speeds coming from volume renderers. To resolve this issue. works in [54, 55] develop light field seld-inspired representations for single-pass pixel rendering. boxel-based scene representations to improve the rendering speed. However, continuous representation inevitably combines with neural networks to implement its complex nature, and this limits rendering speed. As an alternative, 3DGS [13] is designed to construct an explicit radiance field through rasterization, not requiring MLP--ased inference. This method leverages anisotropic 3D Gaussians as the scene representation and proposes differentiable rasterizer to render them by splatting onto the image plane for static scenes.
Both [56] and [57] introduce efficient voxel
2.2 Dynamic Novel View Synthesis
The time domain in dynamic scenes is typically parameterized with Plenoptic function [58]. Classical image-based rendering [21, 22] and novel view synthesis methods using explicit geometry [30, 31] are restricted to limited memory space [21] when they extend to the time dimension. This is because they require additional optimization procedures for crame-by-framt dynamic view synthesis and storage for the parameterized space. Fueled by implicit representations, works in [2, 3, 8, 59, 4. S] handle challenging tasks using neural volume rendering. They learn dynamic scenes by optimizing deformation and canonical fields for object motions [2, 3, 6], using human body priors [60, 59, 61, 62, 63, 64, 65, 66], and decoupling dynamic parts [4, 5, 67, 68, 69, 70, 71, 8, 72]. These methods introduce additional neural fields or learnable variables to represent the time domain, taking more memory usage and rendering time as well.
Temporally extended 3DGS has been considered as feasible solution to dynamic novel view synthesis. work in [73] assigns parameters to 3DGS at each timestamp and imposes rigidity through regularization. Another work in [74] leverages Gaussian probability to model density changes over time to explicitly represent dynamic scenes. However, they require many primitives to capture complex temporal changes. Concurrently, works in [14, 75, 76, 77, 78, 79] utilize MLPs to represent the temporal changes. These methods inherit the drawbacks of dynamic neural representations, resulting in slower rendering speeds. The others in [15, 80] explicitly parameterize the motion of dynamic 3D Gaussians to preserve the rendering speed of 3DGS by predicting their trajectory function. However, they only handle the motion continuous trajectory, and require multiple Gaussians for motions that disappear and reappear due to self-occlusion, increasing memory burden.
In contrast, our key idea for dynamic 3D Gaussian uses keyframes to minimize primitives and to devise a progressive optimization to cope with scenarios where face disappcaring/reappeaiin objects. Thanks to our schemes, we can improve rendering speed, memory efficiency, and achieve impressive performance for dynamic novel view synthesis.
3 Preliminary: 3D Gaussian Splatting
Our model starts from the point-based differentiable rasterization of 3DGS [13]. 3DGS uses three- dimensional Gaussian as geometry primitive, which is composed of position (mean) μ., covariation Σ, density and color The 3D Gaussian is referred to follows:
(1) We need to project the 3D Gaussian onto a 2D plane to render an image. In this process, the approximated graphies pipeline is used to render 2D Gaussians. The covariance matrix E' in camera coordinate is given as follows:
g(x) = e (#-n) Σ ((-na)
J'=JWE
where J is the Jacobian of the affine approximation of the perspective projection and W is a viewing transformation. By skipping the third row and column of E', is approximated to two-dimensional anisotropic Gaussian on the image plane.
3

---

Interpolated frames
Dynamic Static
SfM points
Dynamic [Sec. 4.2]
Gaussian keyframe
Rendering
interpolation [Sec. 4.2)
Set Static
Separation
Prune
Point
Progressive learning (Sec. 4.3]
backracking
Updated motion
Retaim
Static (Sec. 4.1]
Initialization [Sec. 4.3]
B Back prop.
Optimization [Sec. 4.4]
Figure 1: Overview of our method. We first initialize 3D Gaussians as static, modeling their motion linearly. During optimization, dynamic and static objects are separated based on the amount of predicted motion, and the 3D Gaussians between the selected keyframes are interpolated and rendered.
The covariance is a positive semi-definite which can be decomposed into a scale S and a rotation R as:
Σ = RSS R
(3)
Spherical harmonics coefficients are used to represent view- new-deepndent color changes as proposed in [81]. A rendered color from the Gaussian uses point-base a blending similar to NeRF's volume rendering. For the interval between points along ray ô which can obtained from ((x), the color of ray is
N
T. - aall) T: - e -Eitoo
C=
where is the number of visible Gaussians along the ray, i and j denote the order of Gaussians by depth.
4 Methodology
To achieve both memory efficiency and rendering capacity, our scheme is two-fold: (1) Keyframe- based interpolation to span position and rotation of Gaussians over time; (2) Classification of static and dynamic Gaussian. These are described in Sections 4. and 4.2. After that, we introduce our progressive training scheme to handle a variety running times in Section 4.3, and deal with details of the optimization process of our method in Section 4.4. The overview of our method is depicted in Figure 1.
4.1 Static Gaussians
Static Gaussian G. is modeled as the same as the 3DGS model except for its position. G. changes the position linearly over time, which can formulated with the position μ at time as below:
t = [0,
put) x t'd
(5)
where x is a pivot position of G, and is a vector representing the translation, and is the duration of scene. We normalize t with I to prevent d from becoming too large.
4.2 Dynamic Gaussians
The dynamic Gaussian model is based on interpolations keyframes, as visualized Figure Specifically, the state of the dynamic Gaussian Ga an intermediate timestamp is synthesized from adjacent keyframes. In this work, we assume that the keyframe interval is uniform for simplicity. nI, Z, t TT where T is its interval and T is set
The keyframe is defined K. (t of timestamps. Ga acquires position and rotation in quaternion from keyframe information. We use different interpolators with different properties for smooth and continuous motion: CHip using polynomial bases is applied for positions, and a Slerp is used for rotations. We further adapt the Gaussian mixture model for temporal opacity handle changes in the visibility of objects over time.
4.2.1 Cubic Hermite Interpolator for Temporal Position
CHip uses a third-degree polynomial. It is commonly used to model dynamic motions shapes [17]. The interpolator function can defined with third-degree polynomials and four variables: position and tangent vector of the start and end points. On the unit interval [0, 1]. given a start point Po at
= 0 and an end point p1 at L. with start tangent mn = O and an end tangent my at 1, CHip can be defined as:

---

Rendered image
(b) Interpolated dynamic points
Rendered image +
Figure 2: Effectiveness of our keyframe interpolation.
CHip(po, mo. Pl; mi;t) = -2f -36² + 1)po + (t² -222 + t)
(6)
+ + 3t 1p + (t² - t² )m,,
where E [0.
Based on CHip, we compute the position μ of Ga at time t as follows:
((() = CCip(ppp Mm, pa m,+1;;
1 nl
m. Pn+1
Pn-1
Pa42 Pn
where n
m
27
21
where Pn, a position of Gaussian at n-thh keyframe, We use tangent values that is calculated using the position of two neighbor keyframes. This design can reduce additional requirements for storing tangent values at each keyframe, while still keeping the representational power for complex motion.
Other interpolators such as linear interpolation or piecewise cubic Hermite interpolating polynomial can be alternative choices. this work, the cubic Hermit interpolator is selected because we can approximate the complex movements of points without any additional computational cost or storage.
4.2.2 Spherical Linear Interpolation for Temporal Rotation
Slerp is typically used for interpolating rotations [18] because linear interpolation causes a bias problem when it interpolates angular value. On the unit interval (0, 1), given the unit vector wy and 21 which represent rotations at f = and / each, Slerp is defined as follows:
Slerp( #1;t) = sin((1 t) xp + sin
sin [(t)
x1, where t E [o, 1] and cos l = x 31 (8)
sin Ω
Slerp can be directly applied to quaternion rotations since it is independent of quaternions and dimensions. We thus have rotation of intermediate frames in the quaternion of Ga at time without any modification as follows:
t 7
nd
(() = Slerp(rn, To+1;() where n =
1'
where r, is the rotation of Gaussian at n-th keyframe.
4.2.3 Temporal Opacities
Modeling the temporal opacity im- portant because it is directly related to appearing/disappearing objects. A straightforward model for temporal opacity is to directly use a single Gaus- sian. However, there is limitation to model diverse cases only using the sin- gle Gaussian, such sudden emerg- ing/slowly vanishing objects and ob- jects disappearing in videos, as illustrated in Figure 3. We introduce the Gaussian mixture model to handle these situations. Since using too many Gaussians impractical, we approximate the model with two Gaussians. We divide the temporal opacity into three cases: when object appears, the object remains and the object disappears. One Gaussian handles the appearance of the object and the other manages disappearance. The interval between two Gaussians indicates the duration of the object when it is fully visible.
Rosl Estruted
a²
of 01 45
01
47
(a) Single Gaussian (b) Gaussian mixtures
(c) Ours
Figure 3: Comparison between the single Gaussian, Gaussian
mixture, and our model for temporal opacity modeling.
Let Gaussian with smaller mean value be gf and the other is of where de <aG. And, a. DG, as and b9 are the mean and variance of g and the mean and variance of g6 each. The temporal opacity
5

---

of at time is defined as follows:
for t << 0; for 0 - t aj for t t> t>ag.
(t(
1,
(10)
Using a single Gaussian may require multiple points to represent object over long duration. In contrast, our model can handle both the short and long temporal opacity of an object using two Gaussians.
4.3 Training Scheme
Progressive training scheme Our goal is to minimize both memory and computational costs the entire pipeline, including preprocessing, not just reducing the representation of 3DGS model in a dynamic scene. To this end, we adopt progressive training scheme that allows learn over a long duration using only small amount of point clouds obtained from the first frame, which illustrated in Figure 4. To effectively handle objects moving disappearing quickly, our model starts to learn a small part of an input video and gradually increases the video duration. The duration is incremented every specific step and made longer by the interval size.
Interpolation
Progressive
learning
Mittion
Rotation
Expand
Opacity
Keyframe
++1
(+ 2/
Expanding time duration As the time duration increases, the number of keyframes in the dynamic Gaussian obviously increases. We estimate the position and rotation of the Gaussian by linear regression using the last p frames when the number of keyframes increases so that the motion information of the previous frame can be shared with the next keyframe.
Figure 4: Progressive learning of dynamic Gaussians.
Extracting dynamic points from static points We want to separate dynamic points from static points without auxiliary information supervision, such as masks or scribbles [70]. The separation done based on the motion of the static points which modeled to movable, so we select the dynamic points based on the distance they moved. In particular, to avoid biased selection of distant points, we measure the motion in image space, normalizing the translation by the distance between points and the camera at the last timestamp. Therefore, if the distance to a point from the camera at |dd
the last timestamp is X, then the expression is || We sort points by the measured movement and convert the top-1) percent of points (in this work, ?) = is empirically set) into dynamic points. The position of the converted dynamic points is estimated at each keyframe using and d. The rotation made to have the same value in all keyframes, and the opacity is initialized to be visible in all keyframes, We perform the extraction when we increase the duration at specific iterations.
4.4 Optimization
Point backtracking for pruning Since it difficult filter out unnecessary dynamic points in a temporal context, we introduce a way to track errors in the image points. Unlike contemporary works [82] that track points, we let our model track the value on single backward pass. We use two measures, LI distance and SSIM, whose formula is as follows:
ε E. I (1 σ,) x 9k)
(11)
Ex II (1-a,)) a3))
where ze is the measured error in image space, is pixel index, and and are the order of Gaussian by depth which is visible at k-th pixel. The accumulated error Etotal is as follows:
Exotal EVEDE E
(12)
EVED
where D a of training views. We prune the points over Etoral every pre-defined step.
Regularizations and losses We use regularization for large motions on both static and dynamic points. The regularization minimizes ||d]| for static points and ||p.+1 P.lll for dynamic points. The optimization process follows 3DGS, which uses differentiable rasterization based on gradient backpropagation. Both L1 loss and SSIM loss, which measure the error between rendered image and its ground truth, are used.

---

Table Comparison of ours with the comparison methods on Neural 3D Video dataset [7]. Training time: Both preprocessing and the accumulated time of all subsequent training phases. Both the training time and FPS are measured under the same machine with an NVIDIA 4090 GPU for strictly fair comparisons. t: STG is done with an H100 GPU machine due to the memory issue. t: Trained
using a dataset split into 150 frames.
PSNR (dR)
Model
Coffee Martini
Cook Spinach
Cut Roasted Beef
Flame Salmon
Flame Steak
Sear Steak
Average
Size
Training time
NcRFPlayer [72]
31.53
30,56
29.35
31.65
31.93
29.13
30.69
5130
0,05
HyperReel [86]
28.37
32.30
32,92
28.26
32.20
32.57
31.10
360
9
Neural Volumes [87]
NIA
NEA
N/A
22.80
N/A
NIA
22.80
NIA
NIA
NIA
LLFF [88]
NIA
NAA
NIA
23.24
NIA
N/A
23.24
NIA
NIA
NIA
DyNeRF [7]
NIA
N/A
29.58
N/A
29.58
28
0.015
1344
HexPlane [11]
N/A
32.04
32.55
29.47
32.08
32.39
31.71
200
NIA
12
K-Planes [12]
29.99
32,60
31.82
30.44
32.38
32.52
31.63
311
0.3
1.8
Voxelss-- [89]
29.63
32.25
32,40
29.81
32.10
31.34
500
1.3
MixVoxcI-XX [89]
30.39
32.31
32.63
30.60
32.10
32.33
31.73
500
4.6
N/A
Im4D [90]
NIA
N/A
32.58
N/A
N/A
N/A
32.58
NIA
NIA
NIA
4K4D [19]
32.86
Dense COLMAP point cloud inpat
STG* [15]
28.41
32.62
32.53
28.61
33.30
33.40
31,48
107
88.5
5.2
4DGS [74]
28.33
32.93
33 85
29.38
34.03
33.51
32.01
6270
71.4
5.5
4DGaussians [14]
27.34
32.46
32.90
29.20
32.51
32.49
31.15
34
136.9
1.7
Spurse
COLMAP point cloud input
STG* [1S]]
27.71
31.43
28.06
32.17
32.67
30.64
109
101.0
1.3
4DGS [74]
26.51
32.11
31.74
26.93
31.44
32.42
30.19
6057
72.0
4.2
4DGaussians [14]
26.69
31.89
25.88
27.54
28.07
31.73
28.63
34
146.6
1.5
3DGiStream [91]
27.75
28.42
31.67
Ours
28.79
33.23
33.73
29.29
33.91
33.69
32.11
115
120.6
0.6
4.5
Implementation Details
Our codebase is built upon 3DGS [13] and Mip-Splatting [83] and uses almost its hyperparameters. For initialization, experiments only COLMAP [84] point clouds from the first frame. The time interval and initial duration are both set to 10. We increment the duration by its interval every 400 iterations. Both static and dynamic regularization parameters set to 0.0001. We employ the RAdam optimizer [85] for training.
Experiments
In this section, we conduct comprehensive experiments on two real-world datasets, Neural 3D Video [7] and Technicolor dataset [20] in Sections 5.1 and 5.2 each. We follow conventional evaluation protocol in [86, 15], which uses subsequences divided from whole videos. We report PSNR, SSIM and LPIPS values. For SSIM, we use scikit_image library. Here, SSIM, and SSIM, are computed using data_range value of and 2, respectively. We also measure frame-per-second (FPS) for rendering speed, and training time including preprocessing time. To compare the robustness of our method according to initial point clouds, we additionally test contemporary works on sparse point cloud initialization, which uses only the point cloud of the first frame in Sections 5.1 and 5.2. We also visualize the separation static and dynamic points to show that our model can successfully distinguish them in Section 5.3. The ablation study shows the effectiiennes of each component in model Section 5.4.
5.1 Neural 3D Video Dataset
Neural 3D Video dataset [7] provides six sets of multi-view indoor videos, captured with range of
18 to 21 cameras with a 2704 x008x2008 2028 resolution and 300 frames. Following the conventional evaluation protocol, both training and evaluation procedures are performed half the original resolution, and the center camera is held out as novel view for evaluation. For fair comparison, we train all models for all 300 frames including concurrent works, except for STG [15]. NeRFPlayer [72] and HyperReel [86]. For NeRFPlayer and HyperReel, we directly borrow the results from [72, 86]. For STG, it not possible train for all 300 frames due to GPÚ memory issue, so we report the results for only 150 frames, which is the maximum duration running single NVIDIA H100 80GB GPU.
7

---

(n) Ground Truth
(b) 4DGS
(c) STG!
(d) 4DGaussins
(c) Ours
Figure 5: Comparison of our Ex4DGS with other the state-of-the-arr dynamic Gaussian splatting
methods on Neural 3D Video [7] dataset.
Coffee Martini
Flune Steak
Fabien
(b) Rendered static points Figure 6: Visualization of our static and dynamic point separation on Coffee Martini, Flame Steak and Fabien scene in Neural 3D Video [7] and Technicolor [20] datasets.
Rendered dynamic points (d) Rendered stutickdynamic points
As shown in Table 1, our model outperforms most of the contemporary models while maintaining the low computational cost. The example is displayed in Figure 5, which shows that our model produces high-quality rendering results over the comparison methods.
Comparison on sparse conditions We also carry out an experiment check if concurrent methods work well with sparse point cloud initialization, which uses only it for the first frame. We report the result in Table 1. Interestingly, all the concurrent methods yield unsatisfactory results because motions in videos are learned by relying on the point clouds, not temporal changes of objects in the training phase. This implies that they require well-reconstruuttd 3D point clouds for proper initialization, while our method is free from the initial condition.
5.2 Technicolor Dataset
Technicolor light field dataset encompasses video recordings captured using 4 x cam- era array, wherein each camera is synchro- nized temporally, and the spatial resolution 2048x x 1088. Adhering to the methodology introduced in HyperReel [86], we reserve the camera positioned at the intersection of the second row and second column for evalua- tion purposes. Evaluation is conducted on five distinct scenes (Birthday, Fabien, Painter, Theater, Trains) using their original full resolution. We retrain STG [15] using the COLMAP point cloud from the first frame, instead of the point cloud from every frame, for strictly fair comparison.
Table 2: Comparison results on the Technicolor dataset
[20]. t: Trained with sparse point cloud input.
Model
PSNR
SSIM
SSIM,
LPIPS
DyNeRF [7]
31.80
NIA
0.958
0.142
HyperReel [86]
32.73
0.906
N/A
0.109
STG [15]
33.23
0,912
0.960
0.085
4DGS [74]
29.54
0.873
0.937
0.149
4DGaussians [14]
30.79
0.843
0.921
0.178
Ours
33.62
0.916
0.962
0.088
As shown Table 2, Ex4DGS comparable with the second-best model in the sparse COLMAP scenario. Although the Technicolor dataset contains various colorful objects, our model successfully synthesizes the novel view without dense prior or additional parameters. The reason why STG shows the impressive performance is that Technicolor dataset does not have rapid movements.

---

5.3 Separation of Dynamic and Static Points
Ex4DGS has capability to separate static and dynamic points during the learning process. To check how well they are separated, we render them individually. Figure 6 shows the separation result. The static and dynamic points are rendered on both the Neural 3D Video and Technicolor datasets. The results demonstrate that the dynamic points are successfully separated from the static points, even if they are trained in an unsupervised manner. As a result, view-dependent color-changing or reflective objects are also identified as dynamic parts. Furthermore, in Coffee Martini scene, EX4DGS demonstrates the ability to detect dynamic fluid in the transparent glasses. It is also worth highlighting that the same object can have static and dynamic components, as shown in the dog 's legs and head being classified as distinct points in the Flame Steak scene.
5.4 Ablation Studies
We conduct an extensive ablation study to check the effectivvness of the pro- posed technical components in Table 3.
Table 3: Ablation studies of the proposed methods.
Method
PSNR
SSIM;
LPIPS
Size(MB)
w/ Linear position
31.12
0.9385
0.0524
204
We first examine the effectiveness of our interpolation method by changing them into linear models. The results show that linear modeling of the posi- tion and rotation reduces the quality of rendering. Interestingly, using different types of interpolations further exacer- bates the performance. If an equal level of polynomial bases are not assigned to both attributes, one of them falls short of the representational capacity, resulting in overfitting or over-smoothing.
w/o Temporal opacity
31.42
0.9394
0.0521
186
w/ Linear rotation
31.26
0.9392
0.0525
148
w/o Progressive growing
31.02
0,9389
0.0550
168
w/ Linear position&rotation
31.32
0,9394
0.0521
172
wo Regularization
31.37
0.9395
0.0522
174
w/o Dynamic point extraction
28.58
0.9280
0,0756
58
w/o Point backtrucking
31.40
0.9394
0.0529
169
Ours
32.11
0.9422
0.0478
1TS
We also show how our dynamic point extraction affects the rendering quality. As expected, complex motions can only be handled with dynamic point modeling. We then evaluate the efficacy of our temporal opacity modeling, and observe the performance degradation when no temporal opacity changes. Points can only disappear by minimizing the size and hiding back to other Gaussian points, making them act as flutters without being removed properly.
We then check the effectiveness of our progressive growing strategy. Without this strategy, the optimization gets stuck in local minima. This is due our approach using only the point cloud from the first frame, which results in a misalignment with their corresponding objects of future frames.
We evaluate our regularization terms for the temporal dynamics of both static and dynamic points within a scene. As expected, incorporating the additional regularization term into the learning process makes the dynamic scene representations better. This benefit comes from the reduction of accumulated motion errors, preventing the points from moving excessively and locating them at correct positions.
Lastly, we examine the effectiveness of our point backtracking approach for pruning step. As expected, the correct removal of the misplaced points mitigates the errors and leads to the best result.
6 Conclusion
We have proposed a novel parameterization of dynamic 3DGS by explicitly modeling 3D Gaussians' motions. To achieve this, we initially set keyframes and predict their position and rotation changes. Primitive parameters of the 3D Gaussians between keyframes are then interpolated. Our strategy for learning dynamic motions enables us to decouple static and dynamic parts of scenes, opening up more intuitive and interpretable representation in 4D novel view synthesis.
Limitations Although we achieve a memory explicit representation of dynamic scenes, two challenges remain. First, our reconstruction can get stuck in local minima for newly appeared objects that are not initialized with 3D points and have no relevant 3D Gaussians in neighboring frames. This issue could be mitigated by initializing new 3D points with an additional geometric prior such as depth information. Second, as 3DGS suffers from scale ambiguity, training on monocular videos is challenging. This is because every 3D Gaussians are treated as dynamic due to the lack of accurate geometric clues for objects at each timestamp. This challenge can be addressed by incorporating an additional semantic cue information like object mask and optical flow, which account for objects' motions more explicitly.

---

References
[1] B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 1.3
[2] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. 3
[3] Keunhong Park, Utkarsh Sinha, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Bruailn Nerfies: Deformable neural radiance fields. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 1,3
[4] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR). 2021. 1.3
[5] Chen Gao, Ayush Saraf. Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3,
[6] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ri- cardo Martin-Brualla, and Steven M Seitz. Hypernerf: a hiigher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG), 2021. 1,3
[7] Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Ncural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2022. 2, 3, 7, 18, 19
[8] Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. D' 2nerf: Self-supervised decoupling dynamic and static objects from a monocular video. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2022. 1,3
[9] Jia- Wei Liu, Yan-Pei Cao. Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2022.
[10] Lingzhi Li, Zhen Shen, Zhongshu Wang. Li Shen, and Ping Tan. Streaming radiance fields for 3d video synthesis. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2022. 1
[11] Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023. 7, 19
[12] Sara Pridovich-Kele Giacomo Mcanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields space, time, and appearance. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 1, 19
[13] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023. 1,3,7, 15
[14] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie. Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3, 7, 8, 16, 17, 18, 19
[15] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IREF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024. 2, 3, 7, 8. 16, 17, 18, 19
[16] C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendacle, Simon Winder, and Richard Szeliski. High- quality video view interpolation using a layered representation. ACM Transactions on Graphics (TOG), 2004. 2
[17] Richard H Bartels, John C Beatty, and Brian A Barsky. An introduction to splines for use in computer graphics and geometric modeling, Morgan Kaufmann, 1995. 2,4
10

---

[18] Ken Shoemake. Animating rotation with quaternion curves. Proceedings of the 12th annual conference on Computer graphics and interactive techniques., 1985. 2
[19] 7hen Xu, Sida Peng. Haotong Lin, Guangzhao Hc, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei 7hou. 4k4d: Real-time 4d view synthesis 4k resolution. In Proceedings of the IEFF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024. 7.
[20] Neus Sabater, Guillaume Boisson, Benoit Vandame, Paul Kerbiriou, Frederic Babon, Matthieu Hog, Tristan Langlois, Remy Gendrot, Olivier Bureller, Arno Schubert, and Valerie Allie. Dataset and pipeline for multi-view light-field video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW), 2017. 2, 7.8
[21] LEVOY. Light field rendering. In Proceedings of ACM SIGGRAPH, 1996. 2,
[22] Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH, 2001. 2,
[23] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Steren magnification: Learning view synthesis using multiplane images. In Proceedings of ACM SIGGRAPH, 2018.
[24] John Flynn, Michael Broxton, Paul Dehevec, Matthew DuVall, Graham Fyffe, Ryan Overheck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2
[25] Pratul Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely, Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[26] Nima Khademi Kalantari, Ting-Chun Chun Wang, and Ravi Ramamoorthi. Learning-base based view synthesis for light field cameras. In Proceedings of ACM SIGGRAPH, 2016. 2
[27] Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. Neural rerendering in the wild. In Proceedings of the IEEÉ/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019. 2
[28] Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019.
[29] Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. Deep view synthesis from sparse photometric images. In Proceedings of ACM SIGGRAPH, 2019.
[30] Gernot Riegler and Vladlen Koltun. Free view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2020. 2,3
[31] Gernot Riegler and Vladlen Koltun. Stable view synthesis. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2021. 2,
[32] Ronghang Hu, Nikhila Ravi, Alexander C. Berg, and Deepak Pathak. Worldsheet: Wrapping world a 3d sheet for view synthesis from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3
[33] Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-basel differentiable renderer. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2019. 3
[34] Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Samn,, Daniel Vlasic, and William T Freeman. Unsupervised training for 3d morphable model regression. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[35] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer. differentiable renderer for image-based 3d rcasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2019. 3
[36] Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[37] Mijeong Kim, Sconguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neu- ral volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
11

---

[38] Jiawei Yang. Marco Pavone. and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[39] Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV),
2021. 3
[40] Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
[41] Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-seale nerfs for virtual fly- thrtuhooughs.... In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 3
[42] Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR, 2022. 3
[43] Kai Zhang, Gemot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.074 0..9492, 2020.
[44] Junoh Lee, Hyunjun Jung, Jin-Hwi Park, Inhwan Bac, and Hae-Gon Jeon. Geometry-aware projective mapping for unbounded neural radiance fields. Proceedings of the International Conference on Learning Representations (ICLR), 2024. 3
[45] Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3
[46] Bangbang Yang, Chong Bao, Junyi Zeng, Hujum Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In P Proceedings of the European Conference Computer Vision (ECCV), 2022. 3
[47] Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[48] Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Proceedings of the Neural Information Processing Systems (NeurlPS), 2020. 3
[49] Xiuming Zhang, Pratul P Srinivasan, Boyang Deng. Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under unknown illumination. ACM Transactions on Graphics (TOG), 2021. 3
[50] Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[51] Wenqi Yang. Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan- Yee K. Wong, Ps-nerf: Neural inverse rendering for multi-view photometric stereo. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 3
[52] Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real-world arbitrary image collections. In Proceedings of the Neural Information Processing Systems (NewrIPS), 2022. 3
[53] Weicai Ye, Shuo Chen. Chong Bao, Hujun Bao. Mare Pollefeys. Zhaopeng Cui, and Guofeng Zhang. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
[54] Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenhaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proceedings of the Neural Information Procensing Systems (NeurIPS), 2021. 3
[55] Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov. R21: Distilling neural radiance field to neural light field for efficient novel view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV), 2022.
12

---

[56] Alex Yu, Ruilong Li, Matthew Tancik. Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3
[57] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision (ECCV), 2022, 3
[58] Edward H Adelson, James R Bergen, et al. The plenoptic function and the elements of early vision, volume 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology. 1991. 3
[59] Chung-Y Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Gemelmacher-Shlizeeeee Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[60] Hongyi Xu, Thiemo Alldicck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. In Proceedings of the Neural Information Processing Systems (NeurIPS), 2021. 3
[61] Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3
[62] Matthew Loper, Naurcen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael Black, Smpl: skinned multi-person lincar model. ACM Transactions on Graphics (TOG), 2015. 3
[63] Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Instantavatar: Learning avatars from monocular video in 60 seconds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023.
[64] ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully control- lable neural 3d portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition (CVPR), 2022. 3
[65] Yunpeng Bai, Yanbo Fan, Xuan Wang. Yong Zhang. Jingxiang Sun, Chun Yuan, and Ying Shan. High- fidelity facial avatar reconstruction from monocular video with generative priors. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2023. 3
[66] Sida Peng. Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodics. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2021. 3
[67] Wenqi Xian, Jia Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoin video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. 3
[68] Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint video using a layered neural representation. ACM Transactions on Graphics (TOG), 2021. 3
[69] Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In 202/ International Conference on 3D Vision (3DV), 2021. 3
[70] Zhenggi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023. 3, 6
[71] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from single video. In Proceedings of the European Conference on Computer Vision (ECCV), 2022. 3
[72] Liangchen Song, Anpei Chen, Zhong Li. Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 2023. 3, 7, 19
[73] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In Proceedings of the International Conference on 3D Vision, 2024.
13

---

[74] Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In Proceedings of the International Conference on Learning Representations (ICLR), 2023. 3, 7, 8. 16, 17. 18, 19
[75] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
[76] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Se-gs: Sparse- controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3
[77] Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3
[78] Shoukang Hu and Ziwei Liu, Gauhuman: Articulated gaussian splatting from monocular human videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024."
[79] Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang. Xiao Tang. Feng Zhu, and Yuchao Dai. 3d geometry-aware deformable gaussian splatting for dynamic view synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2024. 3
[80] Youtian Lin, Zaozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 3
[81] Sara Fridovich-Ked Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022. 4
[82] Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai,, Lci Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR). 2024.
[83] Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting Alias-free 3d gaussian splatting. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.7
[84] Johannes L Schonberger and Jan-Michael Frahm. Stracturefrrommmmttion revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2016. 7
[85] Liyun Liu, Haoming Jiang, Pengeheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In Proceedings of the International Conference on Learning Representations (ICLR), 2020. 7
[86] Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray ray-conditioned sampling. In Proceedings of the JEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2023. 7, 8, 18, 19
[87] Stephen Lombardi. Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG), 2019. 7, 19
[88] Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng. and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG), 2019. 7, 19
[89] Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafci Song, and Huaping Liu. Mixed neural voxels for fast milti-view vidco synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023. 7, 19
[90] Haotong Lin, Sida Peng, Zhen Xu, Tao Xie, Xingyi Hc, Hujun Bao, and Xiaowei Zhou. High-fidelity and real-time novel view synthesis for dynamic scenes. In Proceedings of SIGGRAPH Asia, 2023. 7, 19
[91] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang. Lei Zhao, and Wei Xing. 3dgstream: On-the-fly -ly training of 3d gaussians for efficient streaming of choto-realisti viee-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024. 7
14

---

A Overview
Within the appendix, we provide additional experiments in Appendix B, additional comparisons in Appendix C and per-scene breakdown of quantitative comparisons in Appendix D.
Additional Experiments
In this section, we conduct experiments further illustrate the behavior of Ex4DGS. Appendix B.1, we present an experiment where changes in color alone are not treated as dynamic points. In Ap- pendix B.2, we demonstrate how Ex4DGS behaves when objects reappear. In Appendix B.3 and Ap- pendix B.4, we present additional ablation studies on various keyframe interval selection and dynamic point conversion rates.
B.1 Without Handling Color Components
Flipeed
(a) Ground Truth
Color change w/o dynamic points
(c) Ours
Rendered - 155f 160l
188r
200r-> Rendered
Figure Comparison between (b) handling color changes without dynamic points and (c) our com- plete model.
Figure 8: Visualization of the rotating decoration in the Technicolor Birthday scene.
We conduct experiments on the Coffee Martini scene from the Neural 3D Video dataset, focusing sce- narios where objects remain stationary but change color. We mask only the moving parts of the objects and train static Gaussians on the remaining regions using the original 3DGS model. The qualitative and quantitative results are presented in Figure 7 and Ta- ble 4. As shown in Figure static points cannot han- dle changes such as shadows. However, Ex4DGS effectively manages these changes using dynamic points, Table "3DGS" denotes the unmodified 3DGS results. "3DGS + Our dynamic" denotes the results when the dynamic regions are replaced with Ex4DGS. Even when only color changes occur without any movement, significant performance loss occurs if these changes are not treated dynamic points.
Table 4: Comparison results between with- out handling color changes and our complete model.
Model
PSNR
SSIM]
LPIPS
3DGS [13]
21.69
0.851
0.126
3DGS +
Our dynamic
26.07
0.891
0.089
Ours
28.79
0.912
0.070
B.2 Reappearing Objects
We visualize how Ex4DGS handles when objects disappear and reappear in Figure 8. Figure illustrates the points associated with the decoration in the Birthday scene from the Technicolor dataset. It shows that the Gaussians corresponding to the part of the decoration that disappears at frame #168 16 and reappears at frame #188 have different distributions after the decoration flips that suggests that reappearing objects are regarded new objects.
B.3 Keyframe Interval Selections
In Table 5, we present results on the effects of keyframe intervals and motion magnitude on Cook Spinach scene from Neural 3D Video dataset. To simulate different motion speeds, we deliberately skip frames in the videas. As shown in Table 5, a keyframe interval of 10 generally yields good results under most conditions. Smaller keyframe intervals tend to perform poorly, especially when the motion speed is low (i.e., fewer frames are skipped). However, as the skipped frame size increases smaller keyframe intervals begin to show better performance. It also shows that the model size decreases as the keyframe interval increases.
15

---

Table 5: Ablation studies of keyframe interval selections and skipped frames.
Skipped frames
Keyframe interval
PSNR
SSIM;
LPIPS
Size(MB)
PSNR
SSIM;
LPIPS Size(MB)
PSNR
SSIM;
LPIPS Size(MB)
31.17
0.948
415
31.81
0.946
32,06
0.952
0.051
314
32.33
0.954
0.049
322
31.81
0.953
0.044
101
10
33.04
0.956
0.041
119
32.65
0.956
0.043
93
31.79
0.953
0.046
74
0.953
50
32.14
0.955
0.046
79
31.93
0.951
0.052
72
30.91
0.949
0.056
73
B.4 Different Dynamic Point Conversion Rates
We experiment with different dynamic point conver- sion rates Cook Spinach scene from Neural 3D Video dataset in Table 6. Our results indicate that the best performance achieved when the extraction percentage is set to 2%. If the percentage is too low, not enough dynamic points will be extracted; con- versely, if it is too high, too many dynamic points may be extracted, leading to overfitting and degraded performance.
Table 6: Ablation studies of dynamic point con- version rate.
Percentage
PSNR
SSIM,
LPIPS
Size(MB)
0.5
32.36
0.955
0.043
103
32.48
0.956
0.044
115
33.04
0.956
0.041
119
32.89
0.955
0.045
31.33
0.954
0.048
C Additional Comparisons
To assess the robustness of EX4DGS, sample different frame intervals from the Technicolor dataset. First, we experiment with occlusion scenarios in Appendix C.I. Next, we present results when new object appears in Appendix C.2. Finally, train extremely long-duration video in Appendix C.3.
C.1 Handling Occlusion
7786
2071
Ground Truth
(b) 4DGS
(c) 4DGaussians
(4) STG
(c) Ours
Figure 9: Qualitative comparison of the repeatedly occluded objects in the Technicolor Train scene over sequence of 100 frames (frame #170 to #269). All models are trained with the point cloud data from the frame #170.
We sample 100 frames (frames #170 #269) from the Train scene in the Technicolor dataset contain- ing occlusions of dynamic objects and compare Ex4DGS with other models. We use the point cloud prior of the first frame, which provides no informa- tion about the reappearing object after the occlusion. We compare the performance of STG, 4DGS and 4D Gaussians models in Table 7 and Figure 9. In these results, the explicit-based models, STG, 4DGS, and ours, perform significantly better. In the case of STG, the dynamic part is not well learned as the frames progress, and while 4DGS can render the dynamic part effectively, struggles with the static part, negatively affecting the overall performance. In particular, 4D Gaussians, being an implicit model, fails to disentangle the static and dynamic components, resulting in missing renderings of the dynamic part. Our model, on the other hand, performs well and effectively learns both static and dynamic parts.
Table 7: Quantitative results of the repeatedly occluded objects in the Technicolor Train scene.
Model
PSNR
SSIM,
LPIPS
STG [15]
32.17
0.940
0.035
4DGS [74]
29.11
0.877
0.119
4DGaussians [14]
23.31
0.657
0.385
Ours
32.24
0.941
0.044
16

---

501
105F
1651
(a) Ciround Truth Truth
(b) 4DGS
ST)
(c) 41 Diusssians
()) Ours
Figure 10: Qualitative comparison of the appearing objects in Technicolor Birthday scene over sequence 120 frames (frame #50 to #169). All models are trained with the point cloud data from the frame #50.
C.2 Handling Newly Appearing Objects
We conduct an experiment to determine whether EX4DGS can learn about newly appearing objects that require the splitting of dynamic components. We sample 120 frames (frame #50 to #169) from the Birthday scene in the Technicolor dataset, during which person appears. All models use a point cloud prior from frame where the person is yet visible. The numerical results are presented Table 8, and the rendered images shown in Figure 10. In contrast With the assertion made in the conclusion,
Table 8: Quantitative results of the appearing
objects in Technicolor Birthday scene.
Model
PSNR
SSIM,
LPIPS
STG [15]
27.62
0.903
0.080
4DGS [74]
28.69
0,907
0.086
4DGaussians [14]
21.51
0.712
0.291
Ours
30.56
0.929
0.051
the result is indeed feasible because Gaussians from neighboring objects can be utilized to facilitate the splitting process, even in the case of newly appearing objects. This is due the effectiveness of the proposed splitting pipeline for static and dynamic Gaussians, which can handle newly appearing objects even when initial Gaussian is provided.
C.3 Extremely Long Duration
(a) Animated video
(b) 1st frame
(c) 167th frame
(d) 333th frame
(c) 500th frame
(I) 667th frame
(g) 833th frame
(h) 1000th frame
(i) 1000th static
(i) 1000th dynamic
Figure 11: Evaluation of the extremely long video Flame Salmon scene in Neural 3D Video dataset.
Best viewed at Adobe Acrobat Reader.
We conduct an experiment using a longer sequence of frames (1,000 frames, 20,000 images in total) on the Flame Salmon scene from the Neural 3D Video dataset. The results are presented in Table 9, and the rendered images are shown Figure 11. The results of this experiment demonstrate that our model is capable of effective learning with reasonable storage requirements, even for extremely long videos. While
Table 9: Quantitative results the extremely long video on Flame Salmon scene in Neural 3D Video dataset.
Model
PSNR
SSIM,
LPIPS
Size(MB)
4DGS [74]
26.26
0.897
0.115
6331
4DGaussians [14]
28.37
0.903
0.097
75
Ours
28.77
0.919
0.076
392
17

---

the 4D Gaussian model produces acceptable results, its performance declines in areas where new objects, such as flames, appear. This indicates that the rendering quality may vary depending on the presence or absence of newly appearing objects, as discussed in Appendix C.I and Appendix C.2.
D Detailed Results
In this section, we report the scene breakdown results of PSNR, SSIM, SSIM,, and LPIPS on the Technicolor dataset and 1, SSIM2 and LPIPS the Neural 3D Video dataset.
Table 10: Per-scene quantitative comparison on Technicolor dataset. t: Trained with sparse point cloud input.
PSNR
Model
Birthday
Fabien
Painter
Theater
Train
Average
DyNeRF [7]
29.20
32.76
35.95
29.53
31.58
31.80
HyperReel [86]
29.99
34.70
35.91
33.32
29.74
32.73
STG [15]]
31.96
34.53
36.47
30.54
32.65
33.23
4DGS [74]
28.01
26.19
33.91
31.62
27.96
29.54
4D Gaussians [14]
30.87
33.56
34.36
29.81
25.35
30.79
Ours
32.38
35.38
36.73
31.84
31.77
33.62
SSIM;
Model
Birthday
Fabien
Painter
Theater
Train
Average
HyperReel [86]
0.922
0.895
0.923
0.895
0.895
0.906
STG* [15]
0.942
0.877
0.923
0.872
0.945
0.912
4DGS [74]
0,902
0.856
0.897
0.869
0.843
0.873
4D Gaussians [14]
0.904
0.854
0.884
0.841
0.730
0.843
Ours
0.943
0.889
0.929
0.880
0.937
0.916
SSIM
Model
Birthday
Fabien
Painter
Theater
Train
Average
DyNeRF [7]
0.952
0.965
0.972
0,939
0.962
0.958
STG' [15]
0.969
0.955
0.970
0.939
0.967
0.960
4DGS [74]
0.944
0.943
0.957
0.940
0.901
0.937
4D
Gaussians [14]
0.950
0.946
0.951
0.925
0.832
0.921
Ours
0.970
0.961
0.972
0,944
0.961
0.962
LPIPS
Model
Birthday
Fabien
Painter
Theater
Train
Average
DyNeRF [7]
0.067
0.242
0.146
0.188
0.067
0.142
HyperReel [86]
0.053
0.186
0.117
0.115
0.072
0.109
STG* [15]]
0.039
0.134
0.097
0.121
0.033
0.085
4DGS [74]
0.089
0.197
0.136
0.155
0.166
0.149
4D Gaussians [14]
0.087
0.186
0.161
0.187
0.271
0.178
Ours
0.044
0.123
0.091
0.129
0.052
0.088
18

---

Table 11: Per-scene quantitative comparison Neural 3D Video dataset. Trained using dataset split into 150 frames.
SSIM1
Model
Coffee Martini
Cook Spinach
Cut Roasted
Flame Salmon
Flame Steak
Scar Steak
Beef
Average
NeRFPlayer [72]
0.951
0.929
0.908
0.940
0.950
0.908
0.931
HyperReel [86]
0.892
0.941
0.945
0.882
0.949
0.952
0.927
Dense COLMAP point cloud input
STG* [15]
0.916
0.952
0.954
0.918
0.960
0.961
0.944
4DGS [74]
NIA
0,980
0.960
N/A
0,970
4DGaussians [14]
0.905
0.949
0.957
0.917
0.954
0.957
0.940
Sparse COLMAP point cloud input
STG! [15]
0.904
0.946
0.946
0.913
0.954
0.955
0.936
4DGS [74]
0.902
0.948
0.947
0.904
0.954
0.955
0.935
4DGaussians [14]
0.893
0.944
0.913
0.896
0.946
0.946
0.923
Ours
0.915
0.947
0.948
0.917
0.956
0.959
0.940
SSIM2
Model
Coffee Martini
Cook Spinach
Cut Roasted
Flame Salmon
Flame
Sear Steak
Beef
Steak
Average
Neural Volumes [87]
N/A
N/A
NIA
0,876
N/A
N/A
0.876
LLFF [88]
N/A
N/A
N/A
0.848
N/A
0,848
DyNeRF [7]
N/A
N/A
0,960
N/A
0,960
HexPlane [11]
N/A
0.970
0,974
0.960
0.978
0.978
0.972
K-Planes [12]
0.953
0.966
0,966
0.953
0,970
0.974
0,964
Mix Voxels-L [89]
0.951
0.968
0,966
0,949
0,971
0.976
0,964
Mix Voxels-X [89]
0,954
0.968
0.971
0.953
0.973
0.976
0.966
Im4D [90]
N/A
N/A
0.970
N/A
N/A
N/A
0.970
4K4D [19]
N/A
N/A 0.972
N/A
N/A
N/A
0.972
Dense COLMAP point cloud input
STG* [15]
0.949
0.974
0.976
0.950
0,980
0.981
0.968
4DGS [74]
N/A
N/A
0.980
0.960
N/A
N/A
0.972
Sparse COLMAP point cloud input
STG* [15]
0.942
0.970
0.971
0,948
0.976
0.977
0.964
4DGS [74]
0.939
0.971
0.970
0,941
0.975
0.976
0.962
4DGaussians [14]
0.934
0.969
0.944
0.937
0.970
0.969
0.954
Ours
0.951
0.976
0.977
0.956
0.980
0.979
0.970
LPIPS
Model
Coffee Martini
Cook Spinach
Cut Roasted Beef
Flame Salmon
Flame
Sear Steak
Steak
Average
NeRFPlayer [72]
0.085
0.113
0.144
0.098
0.088
0.138
0.111
HyperReel [86]
0.127
0.089
0.084
0.136
0.078
0.077
0.096
Neural Volumes [87]
NIA
N/A
N/A
0.295
N/A
N/A
0.295
LLFF [88]
NIA
N/A
N/A
0.235
N/A
N/A
0.235
DyNeRF
N/A
N/A
N/A
0,083
N/A
N/A
0,083
HexPlane [1]]
N/A
0.082
0.080
0.078
0.066
0.070
0.075
Mix Voxels-L [89]
0.106
0.099
0.088
0.116
0.088
0.080
0.096
Mix Voxelsxx [89]
0.081
0.062
0.057
0.078
0.051
0.053
0.064
Dense COLMAP point cloud input
STG [15]
0.069
0.043
0.042
0.063
0.034
0.033
0,047
4DGS [74]
N/A
N/A
0.041
N/A
N/A
N/A
0.055
Sparse COLMAP point cloud inpul
STG* [15]
0.087
0.056
0.060
0.074
0.046
0.046
0.062
4DGS [74]
0.079
0.041
0.041
0.078
0.036
0.037
0.052
4DGaussians [14]
0.095
0.056
0.104
0.095
0.050
0.046
0.074
Ours
0.070
0.042
0.040
0.066
0.034
0.035
0.048
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract concisely outlines the problem definition and proposed methodol-
ogy.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations inherent to the proposed method in the conclusion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-texx system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknow lodgew in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
20

---

Answer: [NA]
Our paper does not address theoretical results, thus it is unrelated to this question.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We provide implementation details for reproduction of results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
We provide the source code in the supplemental materials along with its instructions.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips cc/
http:/guidess/sdseeeeee ceaibbmisssoooooooo cesssonoolio olicy) for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: / //nips. cc/ wtc/public//ee /twp///g/////eeeee/ceecccec.esee htte//ooeSuiiieeee asssonoliicyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide all training details, including the balancing values of loss terms.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification:
We simply report and compare the evaluations without statistical analysis.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
22

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are of range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We provide the information on the computer resources in the Implementation Details section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https: https::/nnet /ttp::///////ppp/eeeeee cccppubllc/.hhccccueeeeee.....ll
Answer: [Yes]
Justification: We follow the code of ethics mentioned in NeurlPS, We only use public datasets and we respect their licenses.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work has no potential positive negative societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: We do not release pretrained models our work focuses on per-scene opti- mization primitives. Additionally, we only use publicly accessible datasets.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use only publicly available datasets, mark their names, and cite the papers that presented them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our work does not involve crowdsourcing experiments nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our work does not involve crowdsoureing experiments nor human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "jgViIFA4XO", "reviewer": "Reviewer_Tqxz", "summary": "The authors propose a fully explicit dynamic Gaussian splatting method, based on keyframe interpolation. The authors separate a dynamic scene into static Gaussians and dynamic Gaussians during training and apply interpolation techniques under temporal explicit representation, including a polynomial basis interpolator for position, a spherical interpolator for rotation, and a simplified Gaussian mixture model for opacity. Additionally, the authors introduce a progressive training scheme and a point-backtracking technique to improve the final convergence. The proposed method was validated on the Neural 3D Video dataset and Technicolor dataset, and exceeds the baselines.", "questions": "1. I am curious about the comparison results between this method and the baselines I mentioned in the weaknesses section.\n\n2. I am curious about the choice of keyframe interval. According to L259, the time interval is set to be 10. I am curious why the authors chose 10 to be the final hyperparameters. Will a different interval number like 5 or 20 affect the final quality?", "limitations": "The limitations have been well discussed by the authors.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-written and easy to understand.\n2. The proposed designs are well-validated. The authors have done a lot of ablation experiments to validate their designs.\n3. As a fully explicit method, the paper shows impressive results regarding to rendering quality, model size, and rendering efficiency.", "weaknesses": "1. Some important baselines are missing, including [1], [2], and [3]. All of them had released their code before the NeuriPS deadline. They have shown better quality than most baselines selected by this paper.\n\n2. Some metrics are missing. I am not sure why in Table 1, the authors only report PSNR. The average LPIPS and SSIM comparison should be included.\n\n3.  The quality improvement is minor compared with some baselines. According to Table 1, the average PSNR is just about 0.2dB higher than some baselines.\n\n[1]3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos\n\n[2]4K4D: Real-Time 4D View Synthesis at 4K Resolution\n\n[3]Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1, Q1]  Thank you for letting us know the relevant papers and we believe that additional comparison with them makes this paper solid. We have thoroughly reviewed these models and will include them in the revised version. You can find the updated results in Table A and we report available values. For 3DGStream [1], we have directly referred to the results from their paper. Similar to 4K4D [2] and Im4D [3], the results have been taken from the STG paper [4].\n\nWhile the works showcase impressive results and offer valuable insights that we deeply respect. However, our approach does have certain advantages over theirs. First, 3DStream [1] does not support random access to time, which means you cannot directly access the desired scene at any arbitrary timestamp. Moreover, 3DStream requires a trained 3DGS model for initialization. \nSecond, both 4K4D [2] and Im4D [3] require background scenes for training. Unfortunately, the N3V dataset does not provide background scenes, our reproduction of Im4D 30.34dB for PSNR. \n\n[W2] Thanks for pointing this out. We attach SSIM and LPIPS in Table B, Table C and Table D. Our results include the models mentioned in W1. We will add it to the revised version. Note that there are two implementation for SSIM metric. SSIM$\\_{1}$ is structural similarity function from _scikit-image_ library and SSIM$_{2}$ is implementation borrowed from 3DGS codebase.\n\n[W3] Other NeRF baselines may have comparable PSNR, but they have drawbacks in rendering/training time or model size. It can also be seen that there is a larger gain in the sparse initial 3D from COLMAP, compared to the other Gaussian Splatting baselines.\n\n[Q2] While final rendering quality depends on the keyframe selection, they are also related to efficiency. Shorter keyframe intervals improve rendering quality because they allow more complex motions to be approximated, but if the keyframe interval is small enough, further reducing the interval is easy to overfit and requires a larger storage size. Therefore, we need to select an appropriately large enough keyframe interval. We report the numerical evaluation of FEDGS with respect to the keyframe interval in Table E.\n\nWe note that the empirical setting can be varied according to properties of datasets. We promise that the evaluation result will be included in the revised version of this paper.\n\n[4] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7Frq5kvcSe", "reviewer": "Reviewer_1Xsu", "summary": "This paper models dynamic scenes using 3DGS, unlike other methods that model dynamic scenes with both implicit and explicit representations. It proposes Fully Explicit Dynamic Gaussian Splatting (FEDGS), a method that models 4D scenes using a purely explicit approach. FEDGS employs a Cubic Hermite Interpolator to predict positions at different times and a Spherical Linear Interpolation for predicting rotations at different times. FEDGS can fit the dynamic scenes with sparse initialization in the N3V dataset without any auxiliary modules or tricks to encode lengthy temporal information.", "questions": "Why the purely explicit approach can bring improvements under sparse COLMAP point cloud conditions is not clearly explained in the paper. I would be pleased to see the author address the issues mentioned in the weakness section, improving the quality and contribution of the paper.", "limitations": "After separating dynamic and static Gaussians, the rendering result of the static parts may experience jitter, which is the main limitation, and it's worth more exploration here.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Modeling 4D scenes with a purely explicit method is rare, and FEDGS can render at 120fps under 157MB, and the training speed is fast.\n2. FEDGS, as a purely explicit method, achieves better results in the case of sparse COLMAP point clouds. Many methods based on 3DGS for dynamic scene reconstruction face challenges with sparse initial point clouds.", "weaknesses": "1. In the supplementary video, compared to other methods, many positions that should be static are moving, causing flickering in objects like hats and tables, and even some Gaussian spheres can be seen rotating. Is this a drawback of the method described in section 4.2.2? This is completely different from the results of previous methods like NeRFPlayer [69] that separate dynamic and static elements, and this part lacks further explanation and clarification.\n2. The most attractive aspect of FEDGS is its dynamic scene reconstruction results under sparse COLMAP point cloud conditions, which may be an advantage of the purely explicit approach. However, the paper does not clearly explain this, and providing more experiments could highlight the advantages of the purely explicit method, enhancing the paper's contribution.\n3. Compared to 4DGaussians, the rendering speed is slower. Is this due to slower predictions of positions and rotations for 3DGS? However, 4DGaussians use hexplane predictions, which should have a higher computational cost than formulas 6 and 9.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] We thank you for your careful comment about the static points. In practice, we observe that there are temporal color and luminance changes even if their position does not change. In this work, our method is designed to handle all the temporal changes as dynamic points. This includes not only the movement of objects, but also changes over time, such as appearances, disappearances, and color changes. In some cases, such as Hats and tables in our supplementary video, there will be temporal shadows or changes in luminance at some timestamp. In these cases, dynamic points account for the light or rotation changes as well, thanks to our spherical harmonics. \nTo validate this, we conduct an experiment with learning from a stationary object to be completely static in Figure B of the uploaded PDF file. We train 3DGS on the coffee martini scene with a mask on dynamic objects which are a person and a glass. We report the results of the test set when 3DGS is used alone and when 3DGS replaces the dynamic parts with our model. The numerical results are shown in Table A. As you can see, learning completely static points, even those that do not change position, leads to poor rendering quality because they cannot handle the other temporal information such as color changes or shadows. Therefore, it is beneficial to treat them as dynamic points.\n\n[W2, Q1] The reason why we have an advantage in the sparse COLMAP condition is that the explicit representation makes the densification algorithm of 3DGS work better. Since the sparse COLMAP condition is trained with fewer points, the empty space needs to be filled with new Gaussians, which are split from existing Gaussian points. In 3DGS, the splitting algorithm works on points that threshold the gradient of the x-/y-direction of the image space. This gradient must be large enough for the Gaussian to be split. However, when using an implicit function, the deformation of the implicit function is optimized first, rather than the gradient of the Gaussian. This interferes with the Gaussians splitting, so implicit models require a more accurate 3D prior. \n\nWe perform several experiments in this regard. The following experiments show results for scenes that are difficult to train. In these experiments, the ability to reconstruct dynamic scenes from sparse COLMAPs implies that these scenes can also be handled by FEDGS.\n\nFirst, we select the 100 frames including the occlusion of dynamic objects in the train scene of the Technicolor dataset and compare FEDGS with others. We use the point cloud prior of the first frame to give no information about the reappearing object after the object is occluded. The results are reported in Table B and the example is displayed in Figure A of the uploaded PDF. \nIn these results, the explicit-based models STG, 4DGS, and Ours perform significantly better. In the case of STG, the dynamic part is not learned well as it goes to the later and later frames, and 4DGS may show that the dynamic part looks good, but it has difficulty in handling the static part which has negative impact on the overall performance. In particular, 4DGaussians, which is an implicit model, fails to disentangle the static and dynamic parts, resulting in missing rendering of the dynamic part. Our model, on the other hand, shows good performance and ability to learn both static and dynamic parts well.\n\nNext, we carry out an experiment to see if FEDGS can learn about newly appearing objects. To handle new objects well, we need to split the dynamic part well. To test this, we select 120 frames from the Technicolor Birthday scene in which a person appears from nowhere. We use the point cloud prior to a frame where the person is invisible. The numerical results are shown in Table C and the rendered images are shown in Figure D of the uploaded PDF file. This result shows that the FEDGS model is beneficial for splitting Gaussians in dynamic scenes and is able to handle newly appearing dynamic objects.\n\nFinally, we conduct an experiment with longer frames (1000 frames, 20,000 images in total) of the flame salmon scene from the N3V dataset. The result is reported in Table D and the rendered image in Figure E of the uploaded PDF file. This result shows that our model can learn well with reasonable storage, even for extremely long videos. \n\n[W3] This is because 4DGaussians can be trained with fewer Gaussians on the N3V dataset than on complex scenes such as the Technicolor dataset. However, 4DGaussians suffers from a huge FPS drop when the minimum number of Gaussians needed is large (i.e. complex scene configurations). To prove it, we conduct an additional experiment on a more complicated scene, Technicolor, and the result is in Table E.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ejsu4gCzKf", "reviewer": "Reviewer_Vpue", "summary": "The paper proposes a new method in the field of novel view synthesis for video input. The authors propose a Gaussian Splatting-based algorithm that introduces a fully explicit representation at keyframes and models interpolation of gaussians (position, rotation, opacity) between the frames. Additionally, the paper proposes to learn scene separation into static and dynamic parts in the progression of training. The method is evaluated on popular video novel view synthesis literature datasets.", "questions": "- HumanRF [1] pays particular attention to how to manage bigger and smaller movements in the video via non-uniform segmentation into manageable chunks - did you notice the high sensitivity of your method to keyframe interval?\n\n- Relating to mentioned more detailed descriptions:\n    - It seems that only the initial set of gaussians is being modified throughout the training. Therefore, how does the appearance of a new object happen? For a big object, there have to be a lot of new gaussians. Also, all appearances in the video have to be modelled with two gaussians. Are there any issues when an object appears and disappears repetitively (e.g. the rotating decoration in the Technicolor Birthday scene)?\n    - Point backtracking for pruning is not fully clear to me. To make sure, does $\\mathcal{D}$ include all training views in all timesteps? Further, what is the pruning rule (is it carried over from the original Gaussian Splatting)?\n    - Regarding linear position change for static gaussians - why is this necessary given the gaussians should be static? Is it to correct tiny movements? Is there no need to allow linear rotation interpolation as well?\n\n- I would encourage authors to check [2] - it seems that this method also models the gaussian representation explicitly in keyframes, however, takes a different approach for motion modelling. Note that this has only just been published at ECCV and does not affect my opinion on this paper, it may just be a worthwhile mention in related work of how the proposed method stands out with respect to others.\n\n[1] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner, *HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion*, ACM Transactions on Graphics (TOG), 2023\n\n[2] Richard Shaw, Jifei Song, Arthur Moreau, Michal Nazarczuk, Sibi Catley-Chandar, Helisa Dhamo, Eduardo Perez-Pellitero, *SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting*, European Conference on Computer Vision, 2024", "limitations": "The authors provided a brief limitations paragraph. I believe the issue with new object appearance is a good mention.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of modelling the scene explicitly in keyframes is certainly an interesting choice - it has the potential to offer a solution to modelling long-range dependencies. In combination with the use of Gaussian Splatting, it is definitely a novel concept.\n- The choice of interpolators seems to be good and well supported through experiments, specifically two-gaussian opacity modelling is a neat way of dealing with objects appearing and disappearing.\n- Separation into static and dynamic Gaussians is a good idea and based on the results and visualisations seems to work well in the proposed methodology. The premise of the majority of the scene being modelled as static was explored in NeRF-based solutions. However, this work provides a novel solution of how to implicitly divide the scene into dynamic and static gaussians.\n- The description of the method is easy to follow. \n- The authors provide a large number of ablation experiments with many components of the methods focused on.\n- The proposed method reports very convincing training times, especially for training on videos.\n- The supplementary video presents the qualitative results in a very clear manner.", "weaknesses": "- I would like to see more detailed descriptions of the approach throughout the paper. Mainly:\n    - How does the temporal opacity modelling relate to the densification of Gaussians?  \n    - More details on keyframe selection would be useful - why fixed step, how is the step chosen.\n    - What is the motivation of static gaussians having linear position interpolation?\n- It would be good to see the same detailed breakdown per scene for the Technicolor dataset. Similarly, the Technicolor experiment shows fewer comparisons to other methods (I would try to include more, at least Gaussian Splatting based ones).\n- The keyframe interpolation naturally draws attention to the method potentially being better at handling big movements or longer videos. Therefore, I think such a comparison is missing in this work. It could be done on the same datasets. Other works compare only short portions of the Technicolor dataset which hides the potential issues with longer videos and bigger movements, however, full scenes are available. Similarly, Neural 3D Video offers a very long sequence for the *Flame Salmon* scene.\n- Even though the ablation is extensive, there could be more insights on the results provided. \n    - Firstly, we are missing the information on what is the ablation performed on - which dataset, one or more scenes.\n    - Is w/o Dynamic point extraction treating everything as static, just with position linear interpolation?\n    - It seems that all the components are highly relevant (removal of one degrades the quality by almost $0.5dB$ at least). Is that a consistent result between multiple runs? Also, is this scene-specific or an average?\n- It would be great if the authors provided some statistical analysis of the results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] Each answer is as follows. The revised version will reflect this description.\n\n1. FEDGS follows the densification algorithm of the original 3DGS. This algorithm accumulates the gradient magnitudes of visible Gaussians (that is, the gradient in the x,y direction of the image space) for a camera during training, selects Gaussians whose gradient magnitudes are greater than a threshold and applies splitting. Here, temporal opacity does not directly involve the densification, but increases or decreases the gradient in 2D direction with respect to visibility over time, allowing the dynamic Gaussian to consider only the gradient in the timestamp in which it is visible.\n\n2. The reason why we fix the keyframe interval is because of optimization issues. While a varying keyframe has advantages in terms of storage (e.g., allocate wider intervals for slow motion and narrower intervals for fast motion), it imposes a high cost on indexing, since the keyframe index of each Gaussian must be searched for every timestamp. On the other hand, if we use fixed timestamps, this cost can be greatly reduced by simple indexing (we can chunk keyframe data as a vector). We choose the fixed step keyframe because we believe that the benefit of reducing the computational cost outweighs the storage wasted by using fixed timestamps, and it is trained to be a reasonable size for the actual size of the storage.\nThe keyframe interval is empirically chosen in consideration of rendering quality and efficiency. Shorter keyframes can handle more complex motion but are also easy to overfit and require more storage. On the other hand, large keyframe intervals often are inadequate to handle complex motion. We add the test result according to a variety of intervals on the N3V dataset in Table A.\n\n3. The reason, why we assume that static Gaussians have linear motions, is to distinguish dynamic points from the other static points. We observe that there is a high correlation between their spatial movements and dynamicity when we train a scene having dynamic motions. We find that when we apply only a linear transformation (i.e., Equation 5) to the 3DGS model, the points with the largest transformation closely coincide with the dynamic part (e.g., selecting 2% of the N3V dataset in order of size matches points corresponding to person). Of course, we could adopt more sophisticated assumptions for the movement, but the linear motion assumption is enough to account for it. We note that the linear motion is normalized based on a distance between each Gaussian and a camera. This aims to impose higher weights on an object close to the camera than a distant one if they have the same distance.\n\n[W2] Thanks for pointing this out. We have added Table B, Table C and Table D with scene breakdown results of PSNR, SSIM and LPIPS respectively for the Technicolor dataset. The reason why there are no results for the other methods is that there are implementation issues because the Technicolor dataset is not officially supported. Nevertheless, we add some baselines to reflect this as best we can.\n\n[W3] We test our model on a 1000-frame scene from the N3V dataset as recommended. We have added an quantitive evaluation in Table E. The rendered images can be found in Figure E of the uploaded PDF file This result shows reasonable performance and memory usage. \n\n[W4] The answers for each item are shown below. We will clarify them in the revised version of this paper.\n1. We report the average results of all scenes in the ablation study N3V dataset.\n2. In w/o dynamic points, the linear motion transformation in Equation 5 is applied to the static points.\n3. As described in W4-1, our ablation reports average over all scenes in N3V dataset. We have added Table F, Table G and Table H with scene breakdown ablation results of PSNR, SSIM and LPIPS respectively for the N3V dataset. We ran the experiment 5 times for each ablation to get multi-run statistics of the ablation result, whose result is reported below. Compared to other cases, ours shows the highest mean PSNR and lowest STD values. Please check it, and please let us know it if you have more questions.\n\n[W5] We report multi-run experiments on all scenes in the N3V dataset which reports only “ours” results of Table F, Table G and Table H. We run our method for 10 times per scene and report mean and STD for each. It shows that our method has lower STD values than other cases, which suggests that each component of our FEDGS contributes to stable performance. Please refer the Table I for detailed results.\n\nThe answers to [Q1, 2] are followed by the next comment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ge2gMEFBQf", "reviewer": "Reviewer_MZ9L", "summary": "The author proposed a Fully Explicit Dynamic Gaussian Splatting method to decrease the training time and memory cost w/o losing the high fidelity. The core of the proposed approach based on the decomposition of the static and dynamic Gaussians during training is to sample dynamic Gaussians at sparse timestamps and then do interpolation to estimate the dynamic Gaussians within each time interval. Also, the proposed backtracking technique is used to prune the useless dynamic Gaussian points. However, there are still some unclear questions. Will consider increasing the rating if the authors can provide reasonable response.", "questions": "1. In lines 238 and 239, why the top 2% points are converted? why consider the distant points as the potential bias?\n2. Can the authors provide more details on how the parameters of keyframes are optimized?\n3. In line 258, what does it mean by 'use only COLMAP point clouds from the first frame'?\n4. What if the movements between key frames are not regular motions?", "limitations": "Please refer to 'Weaknesses'.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. unlike the existing works, the authors attempt to select keyframes and utilize different interpolation techniques on keyframes to model the dynamic Gaussians.\n2. The results on the N3V dataset show that the proposed method achieves comparable or higher performance than the existing methods, while largely reducing the memory and training time cost.\n3. The novel techniques to extract the dynamic points and model the temporal opacity seem very interesting.", "weaknesses": "1. From the results shown in Fig.6 and the supplementary materials, the extracted points from the 'dynamic points extraction' technique seem actually to be the points with color change. I think the results are not very concrete to support the contribution. More experiments in the scenes including objects with large movements might be helpful.\n2. From the proposed opacity modeling technique and the limitations claimed in Sec.6, it seems the proposed method can only work well in such scenes that the objects within the scenes are stable w/o new objects coming in. I would like to know the performance of the proposed method when some new objects come in while progressively processing. It seems that the results do not contain such scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "[W1] The main motivation of separating dynamic points is to allow FESGS to handle temporal changes with either color or displacement variations, so stationary objects but temporally changing colors are also trained as dynamic points. \n\nWe carry out an additional experiment for the case where points with color changes are regarded as static points. We train 3DGS by masking only the moving parts of N3V, and measure the performance gain (To measure overall gain, we replace dynamic parts with ours) on the remaining parts. Please refer to Table A for the detailed results.\nAs shown in the evaluation, if we do not consider the points with color changes as dynamic points, it is hard to obtain satisfactory results.\n\nIn the case of large movements, we conduct an experiment with longer frames on the birthday scene in Figure D of the uploaded PDF file. The rendering result shows that FEDGS detects the movements of objects as well as the color changes. The quantitative result in detail for the large movements will be covered in the next answer.\n\n[W2] We conduct an experiment with different frames of the Technicolor dataset to see this issue. The selected interval is from the timestamp when the person is completely invisible to the timestamp when the bag is placed on the desk. The result is reported in Table B and Figure D of the uploaded PDF file. In this case, the initialized point clouds only use the COLMAP of the first timestamp (no prior point cloud for the person). This result shows that FEDGS can learn about appearing objects on the scene. This is because the densification of 3DGS works well due to the explicit representation that allows splitting from neighboring Gaussians.\n\n[Q1] This is because 2% of the static points with the biggest movement are related to the dynamic part. We note that the 2% is determined empirically, and we observe that  extracting more points causes too many dynamic points and extracting fewer points result in not enough dynamic points. The performance change according to the ratio is shown in Table C.\n\n[Q2] The keyframe is only associated with dynamic points. And, the parameters optimized in each keyframe are about the position and rotation values, and they are optimized by RAdam.\nUsing Equation 7, the CHIP interpolates the position values stored in the four neighboring keyframes for the dynamic points at timestamp $t$. The interpolated position is represented as a $R^3$ vector whose gradient can be obtained by the differential rasterizer of 3DGS. After obtaining the gradient, by Equation 6, the gradient of each keyframe's position can also be obtained by partial difference (we use PyTorch's autograd function) and used to update the position values stored in each keyframe. It is worth noting that thanks to CHIP, FEDGS can also learn the velocity change and momentum value of the dynamic point.\nSimilarly, the keyframe rotation value can also be optimized by Equation 8 and the differential rasterizer in 3DGS. The difference with the keyframe position is that it uses two neighboring keyframes.\n[Q3] This means that for every timestamp, only the image from the earliest timestamp is used. For the N3V dataset, for example, the COLMAP result is provided for the first frame (about 20 images) out of a total of 300 frames. Note that 4DGS models or STGs use all frames (about 6000 images). This prevents COLMAP from taking an excessive computational time to optimize. \n\n[Q4] Irregular motion can be handled by CHIP, given a small enough keyframe interval. CHIP uses the tangent and position of two neighboring keyframes. As shown in Equation. 7, the difference of positions between two adjacent keyframes is used for tangent. The position values are explicitly stored in each keyframe. With the two values, CHIP approximates dynamic motion as a polynomial. If the keyframe is too sparse, it becomes difficult to approximate the motion with CHIP (i.e higher degree polynomials are required), but if it is dense enough, most of the motion can be approximated. This is true even if the motion is velocity-variant or non-linear. This can be seen in the results in Table D. The experiment is performed on cook spinach scene in the N3V dataset. This scene includes a motion that changes direction, accelerates and decelerates. We also include experiments that skip frames to simulate larger movements (the more frames skipped, the larger the movement). These results show that if the keyframe of FEDGS is dense enough, it can be handled (even if the motion is not regular). In general, setting a keyframe interval of 10 is reasonable considering overall performance and storage size.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
First, we would like to thank all the reviewers for giving us the valuable opinions for our paper. All reviewers agree that FEDGS is comparable in terms of performance and efficiency to other models. Reviewers Tqxz and Vpue comment on the clarity of the paper. Reviewers Tqxz, Vpue, and MZ9L highlight the novelty of our fully explicit model. Reviewers Vpue and MZ9L point out that keyframe-based interpolation is interesting, and that treating static and dynamic separately is a good idea. In response to the reviewer's questions, we further analyze our models with various situations, including replacing dynamic part, reappearing object, occluded object, appearance from void, and extremely long duration. We have uploaded  these analyses and additional results to support them as a PDF file. We hope all the reviewers carefully check and give us more comments.

Meta-review: The paper is on creating and reconstructing a dynamic 3DGS model. The idea is interesting but several important weaknesses were initially raised by the reviewers. The discussion clarified most of these weaknesses and an accept consensus was reached. The authors are encouraged to include the contents of the discussion in the revised paper, in particular the additional results provided.
Decision: Accept (poster)