Title: Fully Explicit Dynamic Gaussian Splatting
Venue: NeurIPS 2024
Accepted: yes
Keywords: Gaussian Splatting

Neural Rendering

Computer Vision
Abstract: 3D Gaussian Splatting has shown fast and high-quality rendering results in static scenes by leveraging dense 3D prior and explicit representations. Unfortunately, the benefits of the prior and representation do not involve novel view synthesis for dynamic motions. Ironically, this is because the main barrier is the reliance on them, which requires increasing training and rendering times to account for dynamic motions. 
In this paper, we design Explicit 4D Gaussian Splatting (Ex4DGS).
Our key idea is to firstly separate static and dynamic Gaussians during training, and to explicitly sample positions and rotations of the dynamic Gaussians at sparse timestamps. The sampled positions and rotations are then interpolated to represent both spatially and temporally continuous motions of objects in dynamic scenes as well as reducing computational cost. 
Additionally, we introduce a progressive training scheme and a point-backtracking technique that improves Ex4DGS's convergence. We initially train Ex4DGS using short timestamps and progressively extend timestamps, which makes it work well with a few point clouds. The point-backtracking is used to quantify the cumulative error of each Gaussian over time, enabling the detection and removal of erroneous Gaussians in dynamic scenes. Comprehensive experiments on various scenes demonstrate the state-of-the-art rendering quality from our method, achieving fast rendering of 62 fps on a single 2080Ti GPU.

Fully Explicit Dynamic Gaussian Splatting

Abstract

3D Gaussian Splatting has shown fast and high-quality rendering results in static scenes by leveraging dense 3D prior and explicit representations. Unfortunately, the benefits of the prior and representation do not involve novel view synthesis for dynamic motions. Ironically, this is because the main barrier is the reliance on them, which requires increasing training and rendering times to account for dynamic motions. In this paper, we design Explicit 4D Gaussian Splatting (Ex4DGS). Our key idea is to firstly separate static and dynamic Gaussians during training, and to explicitly sample positions and rotations of the dynamic Gaussians at sparse timestamps. The sampled positions and rotations are then interpolated to represent both spatially and temporally continuous motions of objects in dynamic scenes as well as reducing computational cost. Additionally, we introduce a progressive training scheme and a point-backtracking technique that improves Ex4DGS's convergence. We initially train Ex4DGS using short timestamps and progressively extend timestamps, which makes it work well with a few point clouds. The pointbacktracking is used to quantify the cumulative error of each Gaussian over time, enabling the detection and removal of erroneous Gaussians in dynamic scenes. Comprehensive experiments on various scenes demonstrate the state-of-the-art rendering quality from our method, achieving fast rendering of 62 fps on a single 2080Ti GPU.

1 Introduction

The recent flood of video content has encouraged view synthesis techniques for highly engaging, visually rich content creation to maintain viewer interest. However, even short form videos require a huge time for both data pre-processing, computing frame-wise 3D point clouds, and training time in novel view synthesis for dynamic motions. Furthermore, these techniques must be considered for running on mobile devices which have a limited computing power and storage space. In this aspect, it is crucial to not only achieve photorealistic rendering results, but also reduce the computational cost related to storage, memory and rendering pipeline. To achieve this, the spatio-temporal representation should be explicit and efficient to handle the complexity of dynamic motions in videos.

Recent methods for dynamic view synthesis are typically based on Neural Radiance Fields (NeRF) [1], which uses implicit multi-layer perceptron (MLP) with a combination of 5-DoF spatial coordinates and the additional time axis [2, 3, 4, 5, 6, 7, 8]. The MLP-based methods have shown high-fidelity rendering quality. However, the inevitable cost of decoding the implicit representation makes rendering critically slow. Various methods try to reduce the cost by adapting explicit representation, such as voxel [9, 10] and matrix decomposition [11, 12]. Nevertheless, since NeRF-based approaches require per-pixel ray sampling and dense sampling for each ray, it is hard to achieve real-time and highresolution renderings.

Meanwhile, a promising alternative, 3D Gaussian Splatting (3DGS) [13], has emerged, which achieves photo-realistic rendering results with significantly faster training and rendering speeds.

Unlike NeRF-based approaches, 3DGS exploits fully explicit point-based primitives of 3D and employs a rasterization-based rendering pipeline. Recent advances [14, 15] attempt to extend 3DGS to 4D domain, handling the motion over time by storing the additional transformation information of 3D coordinates or 4D bases. However, these methods can only be trained under a restricted condition with dense point clouds. Moreover, these approaches borrow implicit representations for implementation, which lose the inherent advantage of 3DGS and make real-world applications more difficult. To become a more scalable model for real-world situations, it is important to be trained under more in-the-wild conditions (i.e., sparse point cloud) with more concise representation.

In this paper, we present Explicit 4D Gaussian Splatting (Ex4DGS), a keyframe interpolation-based 4D Gaussian splatting method that works well in a fully explicit on-time domain. Our key idea is to apply interpolation techniques under temporal explicit representation to realize the scalable 3DGS model. Temporal interpolation is a widely used technique in computer graphics [16] that only stores keyframe information in video and determines smooth transitions for the rest of the frames. We select keyframes with sparse time intervals and save the additional temporal information at each keyframe which includes each Gaussian's position, rotation and opacity. This information is fully explicit; it is stored without any encoding process, and continuous motion is calculated to have smooth temporal transitions between adjacent keyframes. Here, we use a polynomial basis interpolator for position, a spherical interpolator for rotation, and a simplified Gaussian mixture model for opacity. Specifically, the polynomial basis of cubic Hermite spline (CHip) [17] is used to effectively avoid overfitting or over-smoothing problems by spanning low-degree polynomials. For rotation, we introduce Spherical Linear Interpolation (Slerp) [18] to do a linear transition over angles. Lastly, we introduce a simplified Gaussian mixture model, which allows temporal opacity to handle the appearance and disappearance of objects.

For further optimization, we reduce the computational cost by isolating dynamic points from static points, and only storing additional temporal information of dynamic points. Here, we aim for this separation to be possible without any additional inputs such as object masks [19]. To tackle this, we introduce motion-based triggers to distinguish static and dynamic points in a scene. We first initialize all Gaussian points in a scene to be static which are assumed to move linearly. During training, static points with large movements are automatically classified as dynamic points. Next, we adopt a progressive training scheme to train our model even under sparse point cloud conditions. The progressive training, starting with a short duration and scaling up, can prevent falling into local minima by reducing the sudden appearance of objects. Lastly, an additional point backtracking technique is introduced to enhance rendering quality. Detecting redundant points in a dynamic scene is challenging because we need to consider all visible timestamps. To measure accumulated errors over time, we apply the point-backtracking technique that can track and prune high-error Gaussians in dynamic scenes.

The effectiveness of our method is validated through experiments on two major real-world video datasets: Neural 3D Video dataset [7] and Technicolor dataset [20]. Experimental results demonstrate that our approach significantly improves rendering results even with sparse 3D point clouds. Furthermore, benefiting from the proposed optimization scheme, our model only requires low storage and memory size without any auxiliary modules or tricks to encode lengthy temporal information. Finally, our model achieves 62 fps on a single 2080Ti GPU on 300-frame scenes with a 1352 × 1014 resolution.

2 Related Works

2.1 Novel View Synthesis

Joint optimization of proxy geometry [32, 33, 34, 35] can help, but direct mesh optimization often gets stuck in local minima due to poor loss landscape conditioning.

In recent years, continuous 3D representation through neural networks has received widespread attention. NeRF [1] is the foundational work that started this trend. It implicitly learns the shape of an object as a density field, which makes optimization via gradient descent more tractable. The robustness of such geometric neural representations enables to reconstruct accurate geometry from a few given images [36, 37, 38, 39], large-scale geometry [40, 41, 42, 43, 44, 45], disentangled textures [46, 47, 48], and material properties [49, 50, 51, 52, 53]. However, one major issue on these neural representations is also slow rendering speeds coming from volume renderers. To resolve this issue, works in [54, 55] develop light field-inspired representations for single-pass pixel rendering. Both [56] and [57] introduce efficient voxel-based scene representations to improve the rendering speed. However, continuous representation inevitably combines with neural networks to implement its complex nature, and this limits rendering speed. As an alternative, 3DGS [13] is designed to construct an explicit radiance field through rasterization, not requiring MLP-based inference. This method leverages anisotropic 3D Gaussians as the scene representation and proposes a differentiable rasterizer to render them by splatting onto the image plane for static scenes.

2.2 Dynamic Novel View Synthesis

The time domain in dynamic scenes is typically parameterized with Plenoptic function [58]. Classical image-based rendering [21, 22] and novel view synthesis methods using explicit geometry [30, 31] are restricted to a limited memory space [21] when they extend to the time dimension. This is because they require additional optimization procedures for frame-by-frame dynamic view synthesis and storage for the parameterized space. Fueled by implicit representations, works in [2, 3, 8, 59, 4, 7, 5] handle challenging tasks using neural volume rendering. They learn dynamic scenes by optimizing deformation and canonical fields for object motions [2, 3, 6], using human body priors [60, 59, 61, 62, 63, 64, 65, 66], and decoupling dynamic parts [4, 5, 67, 68, 69, 70, 71, 8, 72]. These methods introduce additional neural fields or learnable variables to represent the time domain, taking more memory usage and rendering time as well.

Temporally extended 3DGS has been considered as a feasible solution to dynamic novel view synthesis. A work in [73] assigns parameters to 3DGS at each timestamp and imposes rigidity through a regularization. Another work in [74] leverages Gaussian probability to model density changes over time to explicitly represent dynamic scenes. However, they require many primitives to capture complex temporal changes. Concurrently, works in [14, 75, 76, 77, 78, 79] utilize MLPs to represent the temporal changes. These methods inherit the drawbacks of dynamic neural representations, resulting in slower rendering speeds. The others in [15, 80] explicitly parameterize the motion of dynamic 3D Gaussians to preserve the rendering speed of 3DGS by predicting their trajectory function. However, they only handle the motion as a continuous trajectory, and require multiple Gaussians for motions that disappear and reappear due to self-occlusion, increasing memory burden.

In contrast, our key idea for dynamic 3D Gaussian uses keyframes to minimize primitives and to devise a progressive optimization to cope with scenarios where face disappearing/reappearing objects. Thanks to our schemes, we can improve rendering speed, memory efficiency, and achieve impressive performance for dynamic novel view synthesis.

3 Preliminary: 3D Gaussian Splatting

Our model starts from the point-based differentiable rasterization of 3DGS [13]. 3DGS uses threedimensional Gaussian as geometry primitive, which is composed of position (mean) µ , covariation Σ , density σ and color c . The 3D Gaussian is referred to as follows:

We need to project the 3D Gaussian onto a 2D plane to render an image. In this process, the approximated graphics pipeline is used to render 2D Gaussians. The covariance matrix Σ ′ in camera coordinate is given as follows:

where J is the Jacobian of the affine approximation of the perspective projection and W is a viewing transformation. By skipping the third row and column of Σ ′ , it is approximated to two-dimensional anisotropic Gaussian on the image plane.

Prune

Dynamic

[Sec. 4.2]

SfM points

Static [Sec. 4.1]

Progressive learning [Sec. 4.3]

Retain

Rendering

Optimization [Sec. 4.4]

Back prop.

Separation

Updated motion

Point

backtracking

3D Gaussian keyframe

interpolation [Sec. 4.2]

𝐝

Set

Static

Initialization [Sec. 4.3]

Interpolated frames

Static

Dynamic

The covariance is a positive semi-definite which can be decomposed into a scale S and a rotation R as:

Spherical harmonics coefficients are used to represent view-dependent color changes as proposed in [81]. A rendered color from the Gaussian uses point-based α blending similar to NeRF's volume rendering. For the interval between points along ray δ which can be obtained from G ( x ) , the color of ray is

where N is the number of visible Gaussians along the ray, i and j denote the order of Gaussians by depth.

4 Methodology

To achieve both memory efficiency and rendering capacity, our scheme is two-fold: (1) Keyframebased interpolation to span position and rotation of Gaussians over time; (2) Classification of static and dynamic Gaussian. These are described in Sections 4.1 and 4.2. After that, we introduce our progressive training scheme to handle a variety of running times in Section 4.3, and deal with details of the optimization process of our method in Section 4.4. The overview of our method is depicted in Figure 1.

4.1 Static Gaussians

Static Gaussian G s is modeled as the same as the 3DGS model except for its position. G s changes the position linearly over time, which can be formulated with the position µ at time t as below:

where x is a pivot position of G s and d is a vector representing the translation, and l is the duration of a scene. We normalize t with l to prevent d from becoming too large.

4.2 Dynamic Gaussians

The dynamic Gaussian model is based on interpolations of keyframes, as visualized in Figure 2. Specifically, the state of the dynamic Gaussian G d at an intermediate timestamp is synthesized from adjacent keyframes. In this work, we assume that the keyframe interval is uniform for simplicity. The keyframe is defined as K = { t | t = nI, n ∈ Z , t ∈ T } where I is its interval and T is a set of timestamps. G d acquires position µ and rotation in quaternion r from keyframe information. We use different interpolators with different properties for smooth and continuous motion: CHip using polynomial bases is applied for positions, and a Slerp is used for rotations. We further adapt the Gaussian mixture model for temporal opacity to handle changes in the visibility of objects over time.

4.2.1 Cubic Hermite Interpolator for Temporal Position

CHip uses a third-degree polynomial. It is commonly used to model dynamic motions or shapes [17]. The interpolator function can be defined with third-degree polynomials and four variables: position and tangent vector of the start and end points. On the unit interval [0 , 1] , given a start point p 0 at t = 0 and an end point p 1 at t = 1 with start tangent m 0 at t = 0 and an end tangent m 1 at t = 1 , CHip can be defined as:

Keyframe

𝑡𝑡

Keyframe

𝑡𝑡

+

𝐼𝐼

Interpolated frames

Keyframe

𝑡𝑡

Keyframe

𝑡𝑡

+

𝐼𝐼

Interpolated frames

Translation

& Rotation

Translation

Translation

& Rotation

Translation

Based on CHip, we compute the position µ of G d at time t as follows:

where p n is a position of Gaussian at n -th keyframe. We use tangent values that is calculated using the position of two neighbor keyframes. This design can reduce additional requirements for storing tangent values at each keyframe, while still keeping the representational power for complex motion.

Other interpolators such as linear interpolation or piecewise cubic Hermite interpolating polynomial can be alternative choices. In this work, the cubic Hermit interpolator is selected because we can approximate the complex movements of points without any additional computational cost or storage.

4.2.2 Spherical Linear Interpolation for Temporal Rotation

Slerp is typically used for interpolating rotations [18] because linear interpolation causes a bias problem when it interpolates angular value. On the unit interval [0 , 1] , given the unit vector x 0 and x 1 which represent rotations at t = 0 and t = 1 each, Slerp is defined as follows:

Slerp can be directly applied to quaternion rotations since it is independent of quaternions and dimensions. We thus have a rotation of intermediate frames in the quaternion of G d at time t without any modification as follows:

where r n is the rotation of Gaussian at n -th keyframe.

4.2.3 Temporal Opacities

Modeling the temporal opacity is important because it is directly related to appearing/disappearing objects. A straightforward model for temporal opacity is to directly use a single Gaussian. However, there is a limitation to model diverse cases only using the single Gaussian, such as sudden emerging/slowly vanishing objects and ob-

(a) Single Gaussian

(b) Gaussian mixtures

(c)

Ours

𝑎𝑠

𝑜

𝑎 𝑓

𝑜

𝑡

𝜎𝑡

1

0

𝑎𝑠

𝑜

𝑎 𝑓

𝑜

𝑡

𝜎𝑡

1

0

𝑎𝑠

𝑜

𝑎 𝑓

𝑜

𝑡

𝜎𝑡

1

0

Real

Estimated

jects disappearing in videos, as illustrated in Figure 3. We introduce the Gaussian mixture model to handle these situations. Since using too many Gaussians is impractical, we approximate the model with two Gaussians. We divide the temporal opacity into three cases: when an object appears, the object remains and the object disappears. One Gaussian handles the appearance of the object and the other manages disappearance. The interval between two Gaussians indicates the duration of the object when it is fully visible.

Let a Gaussian with a smaller mean value be g o s and the other is g o f where a o s < a o f . And, a o s , b o s , a o f and b o f are the mean and variance of g o s and the mean and variance of g o f each. The temporal opacity

σ t at time t is defined as follows:

Using a single Gaussian may require multiple points to represent an object over a long duration. In contrast, our model can handle both the short and long temporal opacity of an object using two Gaussians.

4.3 Training Scheme

Progressive training scheme Our goal is to minimize both memory and computational costs of the entire pipeline, including preprocessing, not just reducing the representation of 3DGS model in a dynamic scene. To this end, we adopt a progressive training scheme that allows to learn over a long duration using only a small amount of point clouds obtained from the first frame, which is illustrated in Figure 4. To effectively handle objects moving or disappearing quickly, our model starts to learn a small part of an input video and gradually increases the video duration. The duration is incremented every specific step and made longer by the interval size.

Expanding time duration As the time duration increases, the number of keyframes in the dynamic Gaussian obviously increases. We estimate the position and rotation of the Gaussian by linear

𝑡

𝑡 + 𝐼

Motion

Rotation

Progressive

learning

𝑡 + 2𝐼

Expand

Interpolation

Opacity

Keyframe

regression using the last ρ frames when the number of keyframes increases so that the motion information of the previous frame can be shared with the next keyframe.

Extracting dynamic points from static points We want to separate dynamic points from static points without auxiliary information or supervision, such as masks or scribbles [70]. The separation is done based on the motion of the static points which is modeled to be movable, so we select the dynamic points based on the distance they moved. In particular, to avoid biased selection of distant points, we measure the motion in image space, normalizing the translation by the distance between points and the camera at the last timestamp. Therefore, if the distance to a point from the camera at the last timestamp is λ , then the expression is ∥ d ∥ ∥ λ ∥ 2 . We sort points by the measured movement and convert the topη percent of points (in this work, η = 2 is empirically set) into dynamic points. The position of the converted dynamic points is estimated at each keyframe using x and d . The rotation is made to have the same value in all keyframes, and the opacity is initialized to be visible in all keyframes. We perform the extraction when we increase the duration or at specific iterations.

4.4 Optimization

Point backtracking for pruning Since it is difficult to filter out unnecessary dynamic points in a temporal context, we introduce a way to track errors in the image as points. Unlike contemporary works [82] that track points, we let our model track the value on a single backward pass. We use two measures, L1 distance and SSIM, whose formula is as follows:

where q k is the measured error in image space, k is a pixel index, and i and j are the order of Gaussian by depth which is visible at k -th pixel. The accumulated error E total is as follows:

where D is a set of training views. We prune the points over E total at every pre-defined step.

Regularizations and losses We use regularization for large motions on both static and dynamic points. The regularization minimizes ∥ d ∥ for static points and ∥ p n +1 -p n ∥ for dynamic points. The optimization process follows 3DGS, which uses differentiable rasterization based on gradient backpropagation. Both L1 loss and SSIM loss, which measure the error between a rendered image and its ground truth, are used.

4.5 Implementation Details

Our codebase is built upon 3DGS [13] and Mip-Splatting [83] and uses almost its hyperparameters. For initialization, our experiments use only COLMAP [84] point clouds from the first frame. The time interval and initial duration are both set to 10 . We increment the duration by its interval every 400 iterations. Both static and dynamic regularization parameters are set to 0 . 0001 . We employ the RAdam optimizer [85] for training.

5 Experiments

In this section, we conduct comprehensive experiments on two real-world datasets, Neural 3D Video [7] and Technicolor dataset [20] in Sections 5.1 and 5.2 each. We follow a conventional evaluation protocol in [86, 15], which uses subsequences divided from whole videos. We report PSNR, SSIM and LPIPS values. For SSIM, we use scikit_image library. Here, SSIM 1 and SSIM 2 are computed using data_range value of 1 and 2 , respectively. We also measure frame-per-second (FPS) for rendering speed, and training time including preprocessing time. To compare the robustness of our method according to initial point clouds, we additionally test contemporary works on sparse point cloud initialization, which uses only the point cloud of the first frame in Sections 5.1 and 5.2. We also visualize the separation of static and dynamic points to show that our model can successfully distinguish them in Section 5.3. The ablation study shows the effectiveness of each component in our model in Section 5.4.

5.1 Neural 3D Video Dataset

Neural 3D Video dataset [7] provides six sets of multi-view indoor videos, captured with a range of 18 to 21 cameras with a 2704 × 2028 resolution and 300 frames. Following the conventional evaluation protocol, both training and evaluation procedures are performed at half the original resolution, and the center camera is held out as a novel view for evaluation. For a fair comparison, we train all models for all 300 frames including concurrent works, except for STG [15], NeRFPlayer [72] and HyperReel [86]. For NeRFPlayer and HyperReel, we directly borrow the results from [72, 86]. For STG, it is not possible to train for all 300 frames due to a GPU memory issue, so we report the results for only 150 frames, which is the maximum duration running on a single NVIDIA H100 80GB GPU.

(b) 4DGS

(d) 4DGaussians

(c) STG †

(e)

Ours

Fabien

(d) Rendered static&dynamic points

(b) Rendered static points

(c) Rendered dynamic points

As shown in Table 1, our model outperforms most of the contemporary models while maintaining the low computational cost. The example is displayed in Figure 5, which shows that our model produces high-quality rendering results over the comparison methods.

Comparison on sparse conditions We also carry out an experiment to check if concurrent methods work well with sparse point cloud initialization, which uses only it for the first frame. We report the result in Table 1. Interestingly, all the concurrent methods yield unsatisfactory results because motions in videos are learned by relying on the point clouds, not temporal changes of objects in the training phase. This implies that they require well-reconstructed 3D point clouds for proper initialization, while our method is free from the initial condition.

5.2 Technicolor Dataset

Technicolor light field dataset encompasses video recordings captured using a 4 × 4 camera array, wherein each camera is synchronized temporally, and the spatial resolution is 2048 × 1088. Adhering to the methodology introduced in HyperReel [86], we reserve the camera positioned at the intersection of the second row and second column for evaluation purposes. Evaluation is conducted on five distinct scenes (Birthday, Fabien, Painter,

Theater, Trains) using their original full resolution. We retrain STG [15] using the COLMAP point cloud from the first frame, instead of the point cloud from every frame, for strictly fair comparison.

As shown in Table 2, Ex4DGS is comparable with the second-best model in the sparse COLMAP scenario. Although the Technicolor dataset contains various colorful objects, our model successfully synthesizes the novel view without dense prior or additional parameters. The reason why STG shows the impressive performance is that Technicolor dataset does not have rapid movements.

5.3 Separation of Dynamic and Static Points

Ex4DGS has a capability to separate static and dynamic points during the learning process. To check how well they are separated, we render them individually. Figure 6 shows the separation result. The static and dynamic points are rendered on both the Neural 3D Video and Technicolor datasets. The results demonstrate that the dynamic points are successfully separated from the static points, even if they are trained in an unsupervised manner. As a result, view-dependent color-changing or reflective objects are also identified as dynamic parts. Furthermore, in Coffee Martini scene, Ex4DGS demonstrates the ability to detect dynamic fluid in the transparent glasses. It is also worth highlighting that the same object can have static and dynamic components, as shown in the dog's legs and head being classified as distinct points in the Flame Steak scene.

5.4 Ablation Studies

We conduct an extensive ablation study to check the effectiveness of the proposed technical components in Table 3.

We first examine the effectiveness of our interpolation method by changing them into linear models. The results show that linear modeling of the position and rotation reduces the quality of rendering. Interestingly, using different types of interpolations further exacer-

bates the performance. If an equal level of polynomial bases are not assigned to both attributes, one of them falls short of the representational capacity, resulting in overfitting or over-smoothing.

We also show how our dynamic point extraction affects the rendering quality. As expected, complex motions can only be handled with dynamic point modeling. We then evaluate the efficacy of our temporal opacity modeling, and observe the performance degradation when no temporal opacity changes. Points can only disappear by minimizing the size and hiding back to other Gaussian points, making them act as flutters without being removed properly.

We then check the effectiveness of our progressive growing strategy. Without this strategy, the optimization gets stuck in local minima. This is due to our approach using only the point cloud from the first frame, which results in a misalignment with their corresponding objects of future frames.

We evaluate our regularization terms for the temporal dynamics of both static and dynamic points within a scene. As expected, incorporating the additional regularization term into the learning process makes the dynamic scene representations better. This benefit comes from the reduction of accumulated motion errors, preventing the points from moving excessively and locating them at correct positions.

Lastly, we examine the effectiveness of our point backtracking approach for pruning step. As expected, the correct removal of the misplaced points mitigates the errors and leads to the best result.

6 Conclusion

We have proposed a novel parameterization of dynamic 3DGS by explicitly modeling 3D Gaussians' motions. To achieve this, we initially set keyframes and predict their position and rotation changes. Primitive parameters of the 3D Gaussians between keyframes are then interpolated. Our strategy for learning dynamic motions enables us to decouple static and dynamic parts of scenes, opening up more intuitive and interpretable representation in 4D novel view synthesis.

Limitations Although we achieve a memory-efficient explicit representation of dynamic scenes, two challenges remain. First, our reconstruction can get stuck in local minima for newly appeared objects that are not initialized with 3D points and have no relevant 3D Gaussians in neighboring frames. This issue could be mitigated by initializing new 3D points with an additional geometric prior such as depth information. Second, as 3DGS suffers from scale ambiguity, training on monocular videos is challenging. This is because every 3D Gaussians are treated as dynamic due to the lack of accurate geometric clues for objects at each timestamp. This challenge can be addressed by incorporating an additional semantic cue information like object mask and optical flow, which account for objects' motions more explicitly.

References

B Mildenhall, PP Srinivasan, M Tancik, JT Barron, R Ramamoorthi, and R Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020. 1, 3

Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 1, 3

Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 1, 3

Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 1, 3

Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 1, 3

Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: a higher-dimensional representation for topologically varying neural radiance fields. ACM Transactions on Graphics (TOG) , 2021. 1, 3

Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 1, 2, 3, 7, 8, 18, 19

Tianhao Wu, Fangcheng Zhong, Andrea Tagliasacchi, Forrester Cole, and Cengiz Oztireli. Dˆ 2nerf: Self-supervised decoupling of dynamic and static objects from a monocular video. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 1, 3

Jia-Wei Liu, Yan-Pei Cao, Weijia Mao, Wenqiao Zhang, David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 1

Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for 3d video synthesis. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 1

Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 1, 7, 19

Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 1, 7, 19

Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (TOG) , 2023. 1, 3, 7, 15

Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 2, 3, 7, 8, 16, 17, 18, 19

Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 2, 3, 7, 8, 16, 17, 18, 19

C Lawrence Zitnick, Sing Bing Kang, Matthew Uyttendaele, Simon Winder, and Richard Szeliski. Highquality video view interpolation using a layered representation. ACM Transactions on Graphics (TOG) , 2004. 2

Richard H Bartels, John C Beatty, and Brian A Barsky. An introduction to splines for use in computer graphics and geometric modeling . Morgan Kaufmann, 1995. 2, 4

Ken Shoemake. Animating rotation with quaternion curves. Proceedings of the 12th annual conference on Computer graphics and interactive techniques. , 1985. 2, 5

Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 2, 7, 19

Neus Sabater, Guillaume Boisson, Benoit Vandame, Paul Kerbiriou, Frederic Babon, Matthieu Hog, Tristan Langlois, Remy Gendrot, Olivier Bureller, Arno Schubert, and Valerie Allie. Dataset and pipeline for multi-view light-field video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshop (CVPRW) , 2017. 2, 7, 8

MLEVOY. Light field rendering. In Proceedings of ACM SIGGRAPH , 1996. 2, 3

Chris Buehler, Michael Bosse, Leonard McMillan, Steven Gortler, and Michael Cohen. Unstructured lumigraph rendering. In Proceedings of ACM SIGGRAPH , 2001. 2, 3

Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In Proceedings of ACM SIGGRAPH , 2018. 2

John Flynn, Michael Broxton, Paul Debevec, Matthew DuVall, Graham Fyffe, Ryan Overbeck, Noah Snavely, and Richard Tucker. Deepview: View synthesis with learned gradient descent. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 2

Pratul P Srinivasan, Richard Tucker, Jonathan T Barron, Ravi Ramamoorthi, Ren Ng, and Noah Snavely. Pushing the boundaries of view extrapolation with multiplane images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 2

Nima Khademi Kalantari, Ting-Chun Wang, and Ravi Ramamoorthi. Learning-based view synthesis for light field cameras. In Proceedings of ACM SIGGRAPH , 2016. 2

Moustafa Meshry, Dan B Goldman, Sameh Khamis, Hugues Hoppe, Rohit Pandey, Noah Snavely, and Ricardo Martin-Brualla. Neural rerendering in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2019. 2

Inchang Choi, Orazio Gallo, Alejandro Troccoli, Min H Kim, and Jan Kautz. Extreme view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019. 2

Zexiang Xu, Sai Bi, Kalyan Sunkavalli, Sunil Hadap, Hao Su, and Ravi Ramamoorthi. Deep view synthesis from sparse photometric images. In Proceedings of ACM SIGGRAPH , 2019. 2

Gernot Riegler and Vladlen Koltun. Free view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV) , 2020. 2, 3

Gernot Riegler and Vladlen Koltun. Stable view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 2, 3

Ronghang Hu, Nikhila Ravi, Alexander C Berg, and Deepak Pathak. Worldsheet: Wrapping the world in a 3d sheet for view synthesis from a single image. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3

Wenzheng Chen, Huan Ling, Jun Gao, Edward Smith, Jaakko Lehtinen, Alec Jacobson, and Sanja Fidler. Learning to predict 3d objects with an interpolation-based differentiable renderer. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2019. 3

Kyle Genova, Forrester Cole, Aaron Maschinot, Aaron Sarna, Daniel Vlasic, and William T Freeman. Unsupervised training for 3d morphable model regression. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2018. 3

Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2019. 3

Alex Yu, Vickie Ye, Matthew Tancik, and Angjoo Kanazawa. pixelnerf: Neural radiance fields from one or few images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3

Mijeong Kim, Seonguk Seo, and Bohyung Han. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Jiawei Yang, Marco Pavone, and Yue Wang. Freenerf: Improving few-shot neural rendering with free frequency regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Ajay Jain, Matthew Tancik, and Pieter Abbeel. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3

Matthew Tancik, Vincent Casser, Xinchen Yan, Sabeek Pradhan, Ben Mildenhall, Pratul P Srinivasan, Jonathan T Barron, and Henrik Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Haithem Turki, Deva Ramanan, and Mahadev Satyanarayanan. Mega-nerf: Scalable construction of large-scale nerfs for virtual fly-throughs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Jonathan T. Barron, Ben Mildenhall, Dor Verbin, Pratul P. Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. CVPR , 2022. 3

Kai Zhang, Gernot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv:2010.07492 , 2020. 3

Junoh Lee, Hyunjun Jung, Jin-Hwi Park, Inhwan Bae, and Hae-Gon Jeon. Geometry-aware projective mapping for unbounded neural radiance fields. In Proceedings of the International Conference on Learning Representations (ICLR) , 2024. 3

Peng Wang, Yuan Liu, Zhaoxi Chen, Lingjie Liu, Ziwei Liu, Taku Komura, Christian Theobalt, and Wenping Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Bangbang Yang, Chong Bao, Junyi Zeng, Hujun Bao, Yinda Zhang, Zhaopeng Cui, and Guofeng Zhang. Neumesh: Learning disentangled neural mesh-based implicit field for geometry and texture editing. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 3

Petr Kellnhofer, Lars C Jebe, Andrew Jones, Ryan Spicer, Kari Pulli, and Gordon Wetzstein. Neural lumigraph rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3

Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lipman. Multiview neural surface reconstruction by disentangling geometry and appearance. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2020. 3

Xiuming Zhang, Pratul P Srinivasan, Boyang Deng, Paul Debevec, William T Freeman, and Jonathan T Barron. Nerfactor: Neural factorization of shape and reflectance under an unknown illumination. ACM Transactions on Graphics (TOG) , 2021. 3

Kai Zhang, Fujun Luan, Qianqian Wang, Kavita Bala, and Noah Snavely. Physg: Inverse rendering with spherical gaussians for physics-based material editing and relighting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3

Wenqi Yang, Guanying Chen, Chaofeng Chen, Zhenfang Chen, and Kwan-Yee K Wong. Ps-nerf: Neural inverse rendering for multi-view photometric stereo. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 3

Mark Boss, Andreas Engelhardt, Abhishek Kar, Yuanzhen Li, Deqing Sun, Jonathan Barron, Hendrik Lensch, and Varun Jampani. Samurai: Shape and material from unconstrained real-world arbitrary image collections. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2022. 3

Weicai Ye, Shuo Chen, Chong Bao, Hujun Bao, Marc Pollefeys, Zhaopeng Cui, and Guofeng Zhang. Intrinsicnerf: Learning intrinsic neural radiance fields for editable novel view synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023. 3

Vincent Sitzmann, Semon Rezchikov, Bill Freeman, Josh Tenenbaum, and Fredo Durand. Light field networks: Neural scene representations with single-evaluation rendering. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2021. 3

Huan Wang, Jian Ren, Zeng Huang, Kyle Olszewski, Menglei Chai, Yun Fu, and Sergey Tulyakov. R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 3

Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3

Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 3

Edward H Adelson, James R Bergen, et al. The plenoptic function and the elements of early vision , volume 2. Vision and Modeling Group, Media Laboratory, Massachusetts Institute of Technology, 1991. 3

Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman. Humannerf: Free-viewpoint rendering of moving people from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu. H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion. In Proceedings of the Neural Information Processing Systems (NeurIPS) , 2021. 3

Thiemo Alldieck, Hongyi Xu, and Cristian Sminchisescu. imghum: Implicit generative models of 3d human shape and articulated pose. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3

Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black. Smpl: A skinned multi-person linear model. ACM Transactions on Graphics (TOG) , 2015. 3

Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges. Instantavatar: Learning avatars from monocular video in 60 seconds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

ShahRukh Athar, Zexiang Xu, Kalyan Sunkavalli, Eli Shechtman, and Zhixin Shu. Rignerf: Fully controllable neural 3d portraits. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 3

Yunpeng Bai, Yanbo Fan, Xuan Wang, Yong Zhang, Jingxiang Sun, Chun Yuan, and Ying Shan. Highfidelity facial avatar reconstruction from monocular video with generative priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3

Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao. Animatable neural radiance fields for modeling dynamic human bodies. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2021. 3

Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021. 3

Jiakai Zhang, Xinhang Liu, Xinyi Ye, Fuqiang Zhao, Yanshun Zhang, Minye Wu, Yingliang Zhang, Lan Xu, and Jingyi Yu. Editable free-viewpoint video using a layered neural representation. ACM Transactions on Graphics (TOG) , 2021. 3

Vadim Tschernezki, Diane Larlus, and Andrea Vedaldi. Neuraldiff: Segmenting 3d objects that move in egocentric videos. In 2021 International Conference on 3D Vision (3DV) , 2021. 3

Zhengqi Li, Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 3, 6

Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan. Neuman: Neural human radiance field from a single video. In Proceedings of the European Conference on Computer Vision (ECCV) , 2022. 3

Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics , 2023. 3, 7, 19

Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In Proceedings of the International Conference on 3D Vision , 2024. 3

Zeyu Yang, Hongye Yang, Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. In Proceedings of the International Conference on Learning Representations (ICLR) , 2023. 3, 7, 8, 16, 17, 18, 19

Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparsecontrolled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Zhiyin Qian, Shaofei Wang, Marko Mihajlovic, Andreas Geiger, and Siyu Tang. 3dgs-avatar: Animatable avatars via deformable 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Shoukang Hu and Ziwei Liu. Gauhuman: Articulated gaussian splatting from monocular human videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang, Feng Zhu, and Yuchao Dai. 3d geometry-aware deformable gaussian splatting for dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 3

Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. 4

Yiwen Chen, Zilong Chen, Chi Zhang, Feng Wang, Xiaofeng Yang, Yikai Wang, Zhongang Cai, Lei Yang, Huaping Liu, and Guosheng Lin. Gaussianeditor: Swift and controllable 3d editing with gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 6

Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 7

Johannes L Schonberger and Jan-Michael Frahm. Structure-from-motion revisited. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2016. 7

Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, and Jiawei Han. On the variance of the adaptive learning rate and beyond. In Proceedings of the International Conference on Learning Representations (ICLR) , 2020. 7

Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2023. 7, 8, 18, 19

Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG) , 2019. 7, 19

Ben Mildenhall, Pratul P. Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ramamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions on Graphics (TOG) , 2019. 7, 19

Feng Wang, Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , 2023. 7, 19

Haotong Lin, Sida Peng, Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, and Xiaowei Zhou. High-fidelity and real-time novel view synthesis for dynamic scenes. In Proceedings of SIGGRAPH Asia , 2023. 7, 19

Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2024. 7

A Overview

Within the appendix, we provide additional experiments in Appendix B, additional comparisons in Appendix C and per-scene breakdown of quantitative comparisons in Appendix D.

B Additional Experiments

In this section, we conduct experiments to further illustrate the behavior of Ex4DGS. In Appendix B.1, we present an experiment where changes in color alone are not treated as dynamic points. In Appendix B.2, we demonstrate how Ex4DGS behaves when objects reappear. In Appendix B.3 and Appendix B.4, we present additional ablation studies on various keyframe interval selection and dynamic point conversion rates.

B.1 Without Handling Color Components

(b) Color change w/o

dynamic points

(c) Ours

Rendered

Flipped

151f

Rendered

200f

188f

168f

Weconduct experiments on the Coffee Martini scene from the Neural 3D Video dataset, focusing on scenarios where objects remain stationary but change color. We mask only the moving parts of the objects and train static Gaussians on the remaining regions using the original 3DGS model. The qualitative and quantitative results are presented in Figure 7 and Table 4. As shown in Figure 7, static points cannot handle changes such as shadows. However, Ex4DGS

effectively manages these changes using dynamic points. In Table 4, "3DGS" denotes the unmodified 3DGS results. "3DGS + Our dynamic" denotes the results when the dynamic regions are replaced with Ex4DGS. Even when only color changes occur without any movement, significant performance loss occurs if these changes are not treated as dynamic points.

B.2 Reappearing Objects

We visualize how Ex4DGS handles when objects disappear and reappear in Figure 8. Figure 8 illustrates the points associated with the decoration in the Birthday scene from the Technicolor dataset. It shows that the Gaussians corresponding to the part of the decoration that disappears at frame #168 and reappears at frame #188 have different distributions after the decoration flips that suggests that reappearing objects are regarded as new objects.

B.3 Keyframe Interval Selections

In Table 5, we present results on the effects of keyframe intervals and motion magnitude on Cook Spinach scene from Neural 3D Video dataset. To simulate different motion speeds, we deliberately skip frames in the videos. As shown in Table 5, a keyframe interval of 10 generally yields good results under most conditions. Smaller keyframe intervals tend to perform poorly, especially when the motion speed is low (i.e., fewer frames are skipped). However, as the skipped frame size increases smaller keyframe intervals begin to show better performance. It also shows that the model size decreases as the keyframe interval increases.

B.4 Different Dynamic Point Conversion Rates

We experiment with different dynamic point conversion rates on Cook Spinach scene from Neural 3D Video dataset in Table 6. Our results indicate that the best performance is achieved when the extraction percentage is set to 2% . If the percentage is too low, not enough dynamic points will be extracted; conversely, if it is too high, too many dynamic points may be extracted, leading to overfitting and degraded performance.

C Additional Comparisons

To assess the robustness of Ex4DGS, we sample different frame intervals from the Technicolor dataset. First, we experiment with occlusion scenarios in Appendix C.1. Next, we present results when a new object appears in Appendix C.2. Finally, we train on an extremely long-duration video in Appendix C.3.

C.1 Handling Occlusion

(b) 4DGS

(c) 4DGaussians

(d) STG

(e) Ours

175f

207f

We sample 100 frames (frames #170 to #269) from the Train scene in the Technicolor dataset containing occlusions of dynamic objects and compare Ex4DGS with other models. We use the point cloud prior of the first frame, which provides no information about the reappearing object after the occlusion. We compare the performance of STG, 4DGS and 4D Gaussians models in Table 7 and Figure 9. In

these results, the explicit-based models, STG, 4DGS, and ours, perform significantly better. In the case of STG, the dynamic part is not well learned as the frames progress, and while 4DGS can render the dynamic part effectively, it struggles with the static part, negatively affecting the overall performance. In particular, 4D Gaussians, being an implicit model, fails to disentangle the static and dynamic components, resulting in missing renderings of the dynamic part. Our model, on the other hand, performs well and effectively learns both static and dynamic parts.

(b) 4DGS

(e) 4DGaussians

(c) STG

(f)

Ours

50f

165f

105f

C.2 Handling Newly Appearing Objects

We conduct an experiment to determine whether Ex4DGS can learn about newly appearing objects that require the splitting of dynamic components. We sample 120 frames (frame #50 to #169) from the Birthday scene in the Technicolor dataset, during which a person appears. All models use a point cloud prior from a frame where the person is not yet visible. The numerical results are presented in Table 8, and

the rendered images are shown in Figure 10. In contrast With the assertion made in the conclusion, the result is indeed feasible because Gaussians from neighboring objects can be utilized to facilitate the splitting process, even in the case of newly appearing objects. This is due to the effectiveness of the proposed splitting pipeline for static and dynamic Gaussians, which can handle newly appearing objects even when no initial Gaussian is provided.

C.3 Extremely Long Duration

We conduct an experiment using a longer sequence of frames (1,000 frames, 20,000 images in total) on the Flame Salmon scene from the Neural 3D Video dataset. The results are presented in Table 9, and the rendered images are shown in Figure 11. The results of this experiment demonstrate that our model is capable of effective learning with reasonable storage requirements, even for extremely long videos. While

the 4D Gaussian model produces acceptable results, its performance declines in areas where new objects, such as flames, appear. This indicates that the rendering quality may vary depending on the presence or absence of newly appearing objects, as discussed in Appendix C.1 and Appendix C.2.

D Detailed Results

In this section, we report the scene breakdown results of PSNR, SSIM 1 , SSIM 2 , and LPIPS on the Technicolor dataset and SSIM 1 , SSIM 2 and LPIPS on the Neural 3D Video dataset.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract concisely outlines the problem definition and proposed methodology.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the limitations inherent to the proposed method in the conclusion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: Our paper does not address theoretical results, thus it is unrelated to this question.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide implementation details for reproduction of results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provide the source code in the supplemental materials along with its instructions.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We provide all training details, including the balancing values of loss terms.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: We simply report and compare the evaluations without statistical analysis.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide the information on the computer resources in the Implementation Details section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: We follow the code of ethics mentioned in NeurIPS. We only use public datasets and we respect their licenses.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work has no potential positive or negative societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: We do not release pretrained models as our work focuses on per-scene optimization of primitives. Additionally, we only use publicly accessible datasets.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We use only publicly available datasets, mark their names, and cite the papers that presented them.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We do not introduce any new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: Our work does not involve crowdsourcing experiments nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: Our work does not involve crowdsourcing experiments nor human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe author proposed a Fully Explicit Dynamic Gaussian Splatting method to decrease the training time and memory cost w/o losing the high fidelity. The core of the proposed approach based on the decomposition of the static and dynamic Gaussians during training is to sample dynamic Gaussians at sparse timestamps and then do interpolation to estimate the dynamic Gaussians within each time interval. Also, the proposed backtracking technique is used to prune the useless dynamic Gaussian points. However, there are still some unclear questions. Will consider increasing the rating if the authors can provide reasonable response.\n\nSTRENGTHS:\n1. unlike the existing works, the authors attempt to select keyframes and utilize different interpolation techniques on keyframes to model the dynamic Gaussians.\n2. The results on the N3V dataset show that the proposed method achieves comparable or higher performance than the existing methods, while largely reducing the memory and training time cost.\n3. The novel techniques to extract the dynamic points and model the temporal opacity seem very interesting.\n\nWEAKNESSES:\n1. From the results shown in Fig.6 and the supplementary materials, the extracted points from the 'dynamic points extraction' technique seem actually to be the points with color change. I think the results are not very concrete to support the contribution. More experiments in the scenes including objects with large movements might be helpful.\n2. From the proposed opacity modeling technique and the limitations claimed in Sec.6, it seems the proposed method can only work well in such scenes that the objects within the scenes are stable w/o new objects coming in. I would like to know the performance of the proposed method when some new objects come in while progressively processing. It seems that the results do not contain such scenarios.\n\nQUESTIONS:\n1. In lines 238 and 239, why the top 2% points are converted? why consider the distant points as the potential bias?\n2. Can the authors provide more details on how the parameters of keyframes are optimized?\n3. In line 258, what does it mean by 'use only COLMAP point clouds from the first frame'?\n4. What if the movements between key frames are not regular motions?\n\nLIMITATIONS:\nPlease refer to 'Weaknesses'.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new method in the field of novel view synthesis for video input. The authors propose a Gaussian Splatting-based algorithm that introduces a fully explicit representation at keyframes and models interpolation of gaussians (position, rotation, opacity) between the frames. Additionally, the paper proposes to learn scene separation into static and dynamic parts in the progression of training. The method is evaluated on popular video novel view synthesis literature datasets.\n\nSTRENGTHS:\n- The idea of modelling the scene explicitly in keyframes is certainly an interesting choice - it has the potential to offer a solution to modelling long-range dependencies. In combination with the use of Gaussian Splatting, it is definitely a novel concept.\n- The choice of interpolators seems to be good and well supported through experiments, specifically two-gaussian opacity modelling is a neat way of dealing with objects appearing and disappearing.\n- Separation into static and dynamic Gaussians is a good idea and based on the results and visualisations seems to work well in the proposed methodology. The premise of the majority of the scene being modelled as static was explored in NeRF-based solutions. However, this work provides a novel solution of how to implicitly divide the scene into dynamic and static gaussians.\n- The description of the method is easy to follow. \n- The authors provide a large number of ablation experiments with many components of the methods focused on.\n- The proposed method reports very convincing training times, especially for training on videos.\n- The supplementary video presents the qualitative results in a very clear manner.\n\nWEAKNESSES:\n- I would like to see more detailed descriptions of the approach throughout the paper. Mainly:\n    - How does the temporal opacity modelling relate to the densification of Gaussians?  \n    - More details on keyframe selection would be useful - why fixed step, how is the step chosen.\n    - What is the motivation of static gaussians having linear position interpolation?\n- It would be good to see the same detailed breakdown per scene for the Technicolor dataset. Similarly, the Technicolor experiment shows fewer comparisons to other methods (I would try to include more, at least Gaussian Splatting based ones).\n- The keyframe interpolation naturally draws attention to the method potentially being better at handling big movements or longer videos. Therefore, I think such a comparison is missing in this work. It could be done on the same datasets. Other works compare only short portions of the Technicolor dataset which hides the potential issues with longer videos and bigger movements, however, full scenes are available. Similarly, Neural 3D Video offers a very long sequence for the *Flame Salmon* scene.\n- Even though the ablation is extensive, there could be more insights on the results provided. \n    - Firstly, we are missing the information on what is the ablation performed on - which dataset, one or more scenes.\n    - Is w/o Dynamic point extraction treating everything as static, just with position linear interpolation?\n    - It seems that all the components are highly relevant (removal of one degrades the quality by almost $0.5dB$ at least). Is that a consistent result between multiple runs? Also, is this scene-specific or an average?\n- It would be great if the authors provided some statistical analysis of the results.\n\nQUESTIONS:\n- HumanRF [1] pays particular attention to how to manage bigger and smaller movements in the video via non-uniform segmentation into manageable chunks - did you notice the high sensitivity of your method to keyframe interval?\n\n- Relating to mentioned more detailed descriptions:\n    - It seems that only the initial set of gaussians is being modified throughout the training. Therefore, how does the appearance of a new object happen? For a big object, there have to be a lot of new gaussians. Also, all appearances in the video have to be modelled with two gaussians. Are there any issues when an object appears and disappears repetitively (e.g. the rotating decoration in the Technicolor Birthday scene)?\n    - Point backtracking for pruning is not fully clear to me. To make sure, does $\\mathcal{D}$ include all training views in all timesteps? Further, what is the pruning rule (is it carried over from the original Gaussian Splatting)?\n    - Regarding linear position change for static gaussians - why is this necessary given the gaussians should be static? Is it to correct tiny movements? Is there no need to allow linear rotation interpolation as well?\n\n- I would encourage authors to check [2] - it seems that this method also models the gaussian representation explicitly in keyframes, however, takes a different approach for motion modelling. Note that this has only just been published at ECCV and does not affect my opinion on this paper, it may just be a worthwhile mention in related work of how the proposed method stands out with respect to others.\n\n[1] Mustafa Işık, Martin Rünz, Markos Georgopoulos, Taras Khakhulin, Jonathan Starck, Lourdes Agapito, Matthias Nießner, *HumanRF: High-Fidelity Neural Radiance Fields for Humans in Motion*, ACM Transactions on Graphics (TOG), 2023\n\n[2] Richard Shaw, Jifei Song, Arthur Moreau, Michal Nazarczuk, Sibi Catley-Chandar, Helisa Dhamo, Eduardo Perez-Pellitero, *SWinGS: Sliding Windows for Dynamic 3D Gaussian Splatting*, European Conference on Computer Vision, 2024\n\nLIMITATIONS:\nThe authors provided a brief limitations paragraph. I believe the issue with new object appearance is a good mention.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper models dynamic scenes using 3DGS, unlike other methods that model dynamic scenes with both implicit and explicit representations. It proposes Fully Explicit Dynamic Gaussian Splatting (FEDGS), a method that models 4D scenes using a purely explicit approach. FEDGS employs a Cubic Hermite Interpolator to predict positions at different times and a Spherical Linear Interpolation for predicting rotations at different times. FEDGS can fit the dynamic scenes with sparse initialization in the N3V dataset without any auxiliary modules or tricks to encode lengthy temporal information.\n\nSTRENGTHS:\n1. Modeling 4D scenes with a purely explicit method is rare, and FEDGS can render at 120fps under 157MB, and the training speed is fast.\n2. FEDGS, as a purely explicit method, achieves better results in the case of sparse COLMAP point clouds. Many methods based on 3DGS for dynamic scene reconstruction face challenges with sparse initial point clouds.\n\nWEAKNESSES:\n1. In the supplementary video, compared to other methods, many positions that should be static are moving, causing flickering in objects like hats and tables, and even some Gaussian spheres can be seen rotating. Is this a drawback of the method described in section 4.2.2? This is completely different from the results of previous methods like NeRFPlayer [69] that separate dynamic and static elements, and this part lacks further explanation and clarification.\n2. The most attractive aspect of FEDGS is its dynamic scene reconstruction results under sparse COLMAP point cloud conditions, which may be an advantage of the purely explicit approach. However, the paper does not clearly explain this, and providing more experiments could highlight the advantages of the purely explicit method, enhancing the paper's contribution.\n3. Compared to 4DGaussians, the rendering speed is slower. Is this due to slower predictions of positions and rotations for 3DGS? However, 4DGaussians use hexplane predictions, which should have a higher computational cost than formulas 6 and 9.\n\nQUESTIONS:\nWhy the purely explicit approach can bring improvements under sparse COLMAP point cloud conditions is not clearly explained in the paper. I would be pleased to see the author address the issues mentioned in the weakness section, improving the quality and contribution of the paper.\n\nLIMITATIONS:\nAfter separating dynamic and static Gaussians, the rendering result of the static parts may experience jitter, which is the main limitation, and it's worth more exploration here.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a fully explicit dynamic Gaussian splatting method, based on keyframe interpolation. The authors separate a dynamic scene into static Gaussians and dynamic Gaussians during training and apply interpolation techniques under temporal explicit representation, including a polynomial basis interpolator for position, a spherical interpolator for rotation, and a simplified Gaussian mixture model for opacity. Additionally, the authors introduce a progressive training scheme and a point-backtracking technique to improve the final convergence. The proposed method was validated on the Neural 3D Video dataset and Technicolor dataset, and exceeds the baselines.\n\nSTRENGTHS:\n1. The paper is well-written and easy to understand.\n2. The proposed designs are well-validated. The authors have done a lot of ablation experiments to validate their designs.\n3. As a fully explicit method, the paper shows impressive results regarding to rendering quality, model size, and rendering efficiency.\n\nWEAKNESSES:\n1. Some important baselines are missing, including [1], [2], and [3]. All of them had released their code before the NeuriPS deadline. They have shown better quality than most baselines selected by this paper.\n\n2. Some metrics are missing. I am not sure why in Table 1, the authors only report PSNR. The average LPIPS and SSIM comparison should be included.\n\n3.  The quality improvement is minor compared with some baselines. According to Table 1, the average PSNR is just about 0.2dB higher than some baselines.\n\n[1]3DGStream: On-the-Fly Training of 3D Gaussians for Efficient Streaming of Photo-Realistic Free-Viewpoint Videos\n\n[2]4K4D: Real-Time 4D View Synthesis at 4K Resolution\n\n[3]Im4D: High-Fidelity and Real-Time Novel View Synthesis for Dynamic Scenes\n\nQUESTIONS:\n1. I am curious about the comparison results between this method and the baselines I mentioned in the weaknesses section.\n\n2. I am curious about the choice of keyframe interval. According to L259, the time interval is set to be 10. I am curious why the authors chose 10 to be the final hyperparameters. Will a different interval number like 5 or 20 affect the final quality?\n\nLIMITATIONS:\nThe limitations have been well discussed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}