Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting
Authors: Zeyu Yang
Hongye Yang
Zijie Pan
Li Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Novel-view synthesis for dynamic scenes
4D Gaussian
point-based rendering
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime as an entirety and propose to approximate the underlying spatio-temporal 4D volume of a dynamic scene by optimizing a collection of 4D primitives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of a 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, as well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Experiments across various benchmarks, including monocular and multi-view scenarios, demonstrate our 4DGS model's superior visual quality and efficiency.

Published as a conference paper at ICLR 2024
REAL-TIME PHOTOREALISTIC DYNAMIC SCENE REP- RESENTATION AND RENDERING WITH 4D GAUSSIAN SPLATTING
Zeyu Yang, Hongye Yang, Zijie Pan, Li Zhang*
Fudan University
https://fudan-zrg github..hhh..... 1o/ 4d -pw-ggasssssssslllaaaa
ABSTRACT
Reconstructing dynamic 3D scenes from 2D images and generating diverse views over time is challenging due to scene complexity and temporal dynamics. Despite advancements in neural implicit models, limitations persist: (i) Inadequate Scene Structure: Existing methods struggle to reveal the spatial and temporal structure of dynamic scenes from directly learning the complex 6D plenoptic function. (ii) Scaling Deformation Modeling: Explicitly modeling scene element deformation becomes impractical for complex dynamics. To address these issues, we consider the spacetime temporal 4D volume of a dynamic scene by optimizing a collection of 4D primi- tives, with explicit geometry and appearance modeling. Learning to optimize the 4D primitives enables us to synthesize novel views at any desired time with our tailored rendering routine. Our model is conceptually simple, consisting of 4D Gaussian parameterized by anisotropic ellipses that can rotate arbitrarily in space and time, well as view-dependent and time-evolved appearance represented by the coefficient of 4D spherindrical harmonics. This approach offers simplicity, flexibility for variable-length video and end-to-end training, and efficient real-time rendering, making it suitable for capturing complex dynamic scene motions. Ex- periments across various benchmarks, including monocular and mieti-view see- narios, demonstrate our 4DGS model's superior visual quality and efficiency.
entirety and propose to approximate the underlying spatio-
INTRODUCTION
Modeling dynamic scenes from 2D images and rendering photorealistic novel views in real-time is crucial in computer vision and graphics. This task has been receiving increasing attention from both industry and academia because of its potential value wide range of AR/VR applications. Recent breakthroughs, such as NeRF (Mildenhall et al., 2020), have achieved photorealistic static scene rendering (Barron et al., 2021; Verbin et al., 2022). However, adapting these techniques to dynamic scenes is challenging due to several factors. Object motion complicates reconstruction, and temporal scene dynamics add significant complexity. Moreover, real-world applications often capture dynamic scenes as monocular videos, making impractical to train separate static scene representations for each frame and then combine them into dynamic scene model. The central challenge is preserving intrinsic correlations and sharing relevant information across different time steps while minimizing interference between unrelated spacetime locations.
Dynamic novel view synthesis methods can be categorized into two groups. The first group employs structures such as MIPs (Li et al., 2022b) or grids (Wang et al., 2023), including their low-rank de- compositions (Fridovich-Ked et al., 2023; Cao & Johnson, 2023; Attal et al., 2023), learn 6l) plenoptic function without explicitly modeling scene motion. The effectiveness of these methods in capturing correlations across different spatial-temporal locations depends on the inherent character- isties of the chosen data structure. However, they lack flexibility in adapting to the underlying scene motion. Consequently, these methods either suffer from parameter sharing across spatial-tempe
"Lii Zhang (lizhangfd@ fudan.edu.cn) is the corresponding author with School of Data Science, Fudan Uni- versity.

---

Published as a conference paper at ICLR 2024
Sampling
Projection
Dynamic scene
Static scene
Image
Condition
Splatting
4D Gaussians
3D Gaussians
2D Gaussians
Figure Schematic illustration of the proposed 4DGS. This diagram illustrates why our primitive is naturally suitable for representing dynamic scenes. Within our framework, the transi- tion from 4D Gaussian to 2D Planar Gaussian can conceptually correspond to the process where a dynamic scene transforms into 2D image through a temporal orthogonal projection and spatial perspective projection.
locations, leading to potential interference, or they operate too independently, struggling harness the inherent correlations resulting from object motion.
In contrast, another group suggests that scene dynamics are induced by the motion or deformation of consistent underlying representation (Pumarola et al., 2020; Song et al., 2023; Abou-Chakra et al., 2022; Luiten et al., 2024). These methods explicitly learn scene motion, providing the potential for better utilization of correlations across space and time. Nevertheless, they exhibit reduced flexibility and scalability in complex sol-world scenes compared to the first group methods.
To overcome these limitations, this study reformulates the task by approximating scene's 's underly- ing spatio-temporal 4D volume by a set of 4D Gaussians. Notably, 4D rotations enable the Gaussian to fit the 4D manifold and capture scene intrinsic motion. Furthermore, Harmonics as a generalization of Spherical Harmonics for dynamic scenes to model time evolution of appearance in dynamic scenes. This approach marks the first-ever model supporting end-to-end training and real-time rendering of high-resolution, photorealistic novel views in complex dynamic scenes with volumetric effects and varying lighting conditions. Additionally, our proposed repre- sentation is interpretable, highly scalable, and adaptable in both spatial and temporal dimensions.
introduce Spherindrical
Our contributions are as follows: We propose coherent integrated modeling of the space and time dimensions for dynamic scenes by formulating unbiased 4D Gaussian primitives along with dedicated splatting- based rendering pipeline. (ii) The 4D Spherindrical Harmonics of our method is useful and interpretable to model the time evolution of view-dependent color in dynamic scenes. (iii) Extensive experiments on various datasets, including synthetic and real, monocular, and multi- view, demonstrate that our method outperforms all previous methods in terms of visual quality and efficiency. Notably, our method can produce photo-realistic, high-resolution video at speeds far beyond real-time.
2 RELATED WORK
Novel view synthesis for static scenes In recent years, the field of novel view synthesis has re- ceived widespread attention. Mildenhall et (2020) the pioneering work that initiated this trend, suggesting using an MLP to learn the radiance field and employing volumetric rendering to synthe- size images for any viewpoint. However, vanilla NeRF requires querying the MLP for hundreds of points each ray, significantly constraining its training and rendering speed. Some subsequent works have attempted to improve the speed, such as employing well-tailored data structures (Chen et al,, 2022: Sun et al., 2022; Hu et al., 2022; Chen et al., 2023), discarding large MLP (Fridovich-Kad Keil et al., 2022), or adopting hash encodings (Müller et al., 2022). Other works (Zhang et al., 2020; Verbin et al., 2022; Barron al., 2021; 2022; 2023) aim enhance rendering quality by addressing
2

---

Published as a conference paper at ICLR 2024
existing issues in the vanilla NeRF, such as aliassig and reflection. Yet, these methods remain fined to the nuances of differentiable volume rendering. In contrast, Kerbl et al. (2023) introduced 3D Gaussian Splatting, novel framework that possesses the advantages of volumetric rendering approaches-ofeerng high-fidelity view synthesis for complex scenes, while also benefiting from the merits of rasterization approaches, enabling real-time rendering for large-scale scenes. Inspired by this work, in this paper, we further demonstrate that Gaussian primitives are also an excellent representation of dynamic scenes.
Novel view synthesis for dynamic scenes Synthesizing novel views of a dynamic scene at de- sired time from series of 2D captures a more challenging task. The intricacy lies in capturing the intrinsic correlation across different timesteps. This task cannot be trivially regarded as an accumu- lation of novel view synthesis for the static scene of each frame, as such an approach is prohibitively expensive, scales poorly for synthesizing new views at time beyond training data, and inevitably fails when observations at a single frame are insufficient to reconstruct the entire scene. Inspired by the success of NeRF, one research line attempts to learn 6D plenoptic function represented by well-tailored implicit or explicit structure without direct modeling for the underlying motion to ad- dress this challenge (Li et al., 2022b; Fridovich-Ko Keil et al., 2023; Cao & Johnson, 2023; Wang et al., 2023; Attal et al., 2023). However, these methods struggle with the coupling between parameters. An alternative approach explicitly models continuous motion deformation, presuming the dynam- ies of the scene result from the movement or deformation of particular static structures, like particle or canonical fields (Pumarola et al., 2020; Song et al., 2023; Abou-Chakra al., 2022; Luiten al., 2024). Among them, point-based approaches have consistently been deemed promising. Recently, drawing inspiration from 3D Gaussian Splatting, Luiten et al. (2024) represented dynamic scenes with set of simplified 3D Gaussians shared across timesteps and optimized them frame-by-frant With the physically-bass -based priors encoded its regularizations, dynamic 3D Gaussians can be op- timized to represent dynamic scenes faithfully given its multi-view captures, and achieve long-term tracking by exploiting dense correspondences across timesteps.
Dynamic 3D Gaussians Recently, there have been substantial efforts (Chen & Wang, 2024) in ex- tending 3D Gaussian Splatting into dynamic scenes. Beyond the pioneer work Luiten et al. (2024) mentioned above, Yang et al. (2023); Wu et al. (2023); Liang et al. (2023) propose to model the geometry and dynamic of scenes by the joint optimization of Gaussians in canonical space and a deformation field. (Kratimenos al., 2023) encourages locality and rigidity between points by fac- torizing the motion in the scene into few neural trajectories. These works ingeniously incorporate a prior of topological invariance into their representations, making them well-suited for reconstructing dynamic scenes from monocular videos. However, they assume that dynamic scenes are generated by a fixed set of 3D Gaussians and the elements composing the scene are always visible. In contrast, by formulating a novel 4D scene primitive, we discard their underlying assumptions and circum- vent the need to maintain ambiguous and complex tracking relationships, thereby facilitating a more flexible and versatile approach to handling complex scenes in real-world applications.
3 METHOD
We propose a novel photorealistic scene representation tailored for modeling general dynamic scenes. In this section, we will delineate each component of and the corresponding optimiza- tion process. In Section 3.1, we will begin by reviewing 3D Gaussian Splatting (Kerbl et al., 2023) from which our method inspired. In Section 3.2, we detail how our 4D Gaussian represents dynamic scenes and synthesizes novel views. An overview is shown in Figure 2. The optimization framework will be introduced in Section 3.3.
3.1 PRELIMINARY: 3D GAUSSIAN SLATTING
3D Gaussian Splatting (Kerbl et al., 2023) employs anisotropic Gaussian to represent static 3D scenes. Facilitated by a well-tailored GPU-friendly rasterizer, this representation enables real-time synthesis of high-fidelity novel views.
Representation of 3D Gaussians In 3D Gaussian Splatting, a scene is represented as a cloud of 3D Gaussians. Each Gaussian has theoretically infinite scope and its influence on given spatial
3

---

Published as a conference paper at ICLR 2024
P(O)
Maginal Time
cdd,t)
Time-evolver evolved & View-dependenn Color
4D Gaussians ,,,,
3D Gaussians px,
Planar Gaussians
puuvv0
2(u, = Σ aip (u, la ())
Figure 2: Rendering pipeline of our 4DGS. Given time & and view I, each 4D Gaussian is first decomposed into conditional 3D Gaussian and a marginal ID Gaussian. Subsequently, the conditional 3D Gaussian is projected to a 2D splat. Finally, we integrate the planar conditional Gaussian, ID marginal Gaussian, and time-evolving deeeddependent color to render the view
position x e R3 defined by an unnormalized Gaussian function:
p(a|u, Σ) = e- (((--)"s="(<-A))
(1)
where μ E IR3 is its mean vector, and E R3x3 an anisotropic covariance matrix. In the Ap- pendix, we will show that it holds desired properties of normalized Gaussian probability density function critical for our methodology. i.e., the unnormalized Gaussian function of a multivariate Gaussian be factorized the production of the unnormalized Gaussian functions of its condi- tion and margin distributions. Hence, for brevity and without causing misconceptions, we do not specifically distinguish between equation I and its normalized version in subsequent sections.
In Kerbl et al. (2023), the mean vector μe of a 3D Gaussian is parameterized as (Jas Mys Pz), and the covariance matrix E is factorized into scaling matrix S and a rotation matrix R as Σ RSST RT, Here S is summarized by its diagonal elements S = diag(sz (22 Sy: $z), whilst R is constructed from a unit quaternion q. Moreover, 3D Gaussian also includes set of coefficients of spherical harmonics (SH) for representing view-dependent color, along with an opacity a.
All of the above parameters can be optimized under the supervision of the rendering loss. During the optimization process, 3D Gaussian Splatting also periodically performs densification and pruning on the collection of Gaussians to further improve the geometry and rendering quality.
Differentiable rasterization via Gaussian splatting In rendering, given pixel (u, v) in view I with extrinsic matrix E and intrinsic matrix K, its color I(u, v) can computed by blending visible 3D Gaussians that have been sorted according to their depth, as described below:
N
i-1 II ---:p, ez 2d )dd))aa)
I(u, Ep.uu vi 2d Edd )a, (;,c;(d)
(2)
==1
where denotes the color of the i-th visible Gaussian from the viewing direction d, represents its opacity, and (u, v)) the probability density of the i-th Gaussian at pixel (u, v)).
To compute p; (u, v) in the image space, we linearize the perspective transformation as in Zwicker al. (2002); Kerbl et al. (2023). Then, the projected 3D Gaussian can approximated by a 2D Gaussian. The mean of the derived 2D Gaussian is obtained as:
(4 Proj E,g|E, K)
(3)
where Proj (: (,EE, K) denotes the transformation from the world space to the image space given the intrinsic and extrinsic E. The covariance matrix is given by
I = (JEEET )1:211:2,
(4)
where J is the Jacobian matrix of the perspective projection.
4

---

d Published as a conference paper at ICLR 2024
3.2 4D GAUSSIAN FOR DYNAMIC SCENES
Problem formulation and 4D Gaussian splatting, To extend the formulation of Kerbl et al. (2023) for modeling dynamic scenes, reformulation necessary. In dynamic scenes, pixel un- der view Z can no longer be indexed solely by pair of spatial coordinates (u, v) in the image plane: But an additional timestamp comes into play and intervenes. Formally this formulated by extending equation as:
N I(u, U. eu,
i-1 II
- ps v, ()))).
(5)
j=1
Note that (u, U. 8) can be further factorized as a product of a conditional probability pi(u, u/t) and marginal probability (t) at time i, yielding:
N
i-1
I(u, 1,() 1)= = Σ p:(t)miu, ol (.) (d)
II - Ps 1y(0ps44
(6)
==1
Let the underlying pr 3D Gaussian, y, 2, be a 4D Gaussian. As the conditional distribution p(a, y, a|t) is also similarly derive p(u, u(t) planar Gaussian whose mean and covariance matrix are parameterized by equation 3 and equation 4, respectively.
Subsequently, it comes the question of how to represent 4D Gaussian. A natural solution is that we adopt a distinct perspective for space and time, that is, considering (x, y, =) and / are independent of each other, i.e., (3 y, z|{) (ar, z). Under this assumption, equation 6 can be implemented by adding an extra ID Gaussian (t( into the original 3D Gaussians (Kerbl et al., 2023). This design can be viewed as imbuing 3D Gaussian with temporal extension, or weighting down its opacity when the rendering timestep is away from the expectation of (t).. However, we show later on that this approach can achieve a reasonable fitting of 4D manifold but is difficult to capture the underlying motion of the scene (see "No-4DRot" in Table 3).
Representation of 4D Gaussian To address the mentioned challenge, we suggest treat time and space dimensions equally by formulating a coherent integrated 4D Gaussian model. Similar Kerbl et al. (2023), we parameterize its covariance matrix E the configuration 4D ellipsoid for easing model optimization:
Σ = RSST R²,
(7)
where 5 is a scaling matrix and R is a 4D rotation matrix. Since S is diagonal, it can be completely inscribed by its diagonal elements S diag(sa, 82, 82). On the other hand, rotation in 4D Euclidean space can be decomposed into a pair of isotropic rotations, cach of which can be represented by a quaternion.
Specifically, given Qi = (a, b, c, d) and g. (p, q, r, s) denoting the left and right isotropic rotations respectively, R can be constructed by:
a
b -c
-b
-d
p
-
-r
-8
= R R R(g)
g
p
-r
(8)
C
a
r
p
9
d
b
-9
p
The mean of a 4D Gaussian can be represented by four scalars as (Jar, Pys P2, (4.).) Thus far we arrive complete representation of the general 4D Gaussian.
Subsequently, the conditional 3D Gaussian can be derived from the properties of the multivariate Gaussian with:
Payzlt = P13 E1:3,4 (4),
Exgzit = E13.1: - Σ13, -4,4 E1,13
Since (x y, 2|t) is 3D Gaussian, Pu,u, ult) in equation 6 can be derived in the same way as in equation and equation Moreover, the marginal (t( is also a Gaussian in dimemniinio
(10) So far have comprehensive implementation of equation 6. Subsequently, can adapt the highly efficient tile-based rasterizer proposed Kerbl et al. (2023) to approximate this process, through considering the marginal distribution pt()) when accumulating colors and opacities.
p(t) / (4: p4: Σ4.4)
5

---

Published as a conference paper at ICLR 2024
4D spherindrical harmonics The view-dependent color (d) in equation 6 is represented by a series of SH coefficients in the original 3D Gaussian Splatting (Kerbl et al., 2023). To more faithfully model the dynamic scenes of real world, we must enable appearance variation with varying viewpoints and also their colors to evolve over time.
Leveraging the flexibility of framework, straightforward solution is to directly different Gaussians to represent the same point at different times. However, this approach leads to duplicated and redundant representation of identical objects, making challenging to optimize. Instead we choose to exploit 4D extension of the spherical harmonics (SH) that directly represents the time evolution of appearance of each Gaussian. The color in equation 6 could then be manipulated with e; (d, t), where d = (0, Φ) is the normalized view direction under spherical coordinates and t is the time difference between the expectation of the given Gaussian and the viewpoint.
Inspired by studies on head-related transfer function, we propose to represent c; c;(d, t) the com- bination of series of 4D spherindrical harmonics (4DSH) which are constructed by merging SH with different ID-basis functions. For computational convenience, adopted ID-basis functions. Consequently, 4DSH can be expressed as:
use the Fourier series as the
2 T
(t, cos
Y1" (1"((0,)))
(11)
where Y is the 3D spherical harmonics. The index 1 2 O denotes its degree, and m is the order satisfying -1 S m S I. The index n is the order of the Fourier series. The 4D spherindrical harmonics form an orthonormal basis in the spherindrical coordinate system.
3.3 TRAINING
Following 3D Gaussian Splatting (Kerbl et al., 2023), we conduct interleaved optimization and density control during training. is worth highlighting that our optimization process is entirely end-to-end, capable of processing entire videos, with the ability to sample at any time and view, as opposed to the traditional frame-by-frame or multi-stage training approaches.
Optimization In optimization, we only use the rendering loss as supervision. In most scenes, com- bining the representation introduced above with the default training schedule as in Kerbl et al. (2023) is sufficient yield satisfactory results. However, in some scenes with more dramatic changes, observe issues such as temporal flickering and jitter. We consider this may arise from suboptimal sampling techniques. Rather than adopting the prior regularization, we discover that straightforward batch sampling in time turns out to be superior, resulting in more seamless and visually pleasing appearance of dynamic visual contents.
Densification in spacetime In terms of density control, simply considering the average mag- nitude of view space position gradients is insufficient assess under-reconstrructin and over- reconstruction over time. To address this, we incorporate the average gradients of br as an addi- tional density control indicator. Furthermore, in regions prone to reonssscontttttinnn we employ joint spatial and temporal position sampling during Gaussian splitting.
4 EXPERIMENTS
In this section, we present comparisons with state-of-the-art methods on two well-established datasets for dynamic scene novel view synthesis: Plenoptic Video dataset (Li et al., 20226) and D-NeRF dataset (Pumarola et al., 2020). Additionally, we perform ablation studies to gain insights into our method and illustrate the efficacy of key design decisions. Video results and more use cases can be found in our supplementary material.
4.1 EXPERIMENTAL SETUP
4.1.1 DATASETS
Plenoptic Video dataset (Li et al., 2022b) comprises six real-world scenes, each lasting ten sec- onds. For each scene, view is reserved for testing while other views are used for training. To

---

Published as a conference paper at ICLR 2024
Table I: Comparison with the state-of-the-art methods on the Plenoptic Video benchmark. 1: Only report the result on the scene flames salmon. 2: Only report SSIM instead of MS-SSIM like others. 9; Measured by ourselves using their official released code. 4; Results on Spinach, Beef, and Steak scenes.
Method
PSNR
DSSIM
LPIPS
FPS
Plenoptic Video (real, mulri-view)
Neural Volumes (Lombardi et al., 2019)*
22.80
0.062
0.295
LLFF (Mildenhall al., 2019)!
23.24
0.076
0.235
DyNeRF (Li et al., 2022b)
29.58
0.020
0.099
0.015
HexPlane (Cao & Johnson, 2023)
31.70
0.014
0.075
0.56
K- exppessexxpliitt (Fridovich- Keil al., 2023)
30.88
0.020
0.23
K- K-Planes-hybrid (Fridovich-Keill et al., 2023)
31.63
0,018
MixVoxels-L. (Wang et al., 2023)
30.80
0.020
0.126
16.7
StreamRF (Li et al., 2022a)²
29.58
8.3
NeRFPlayer (Song et al., 2023)
30.69
0.035
0.111
0.045
HyperReel (Attal al., 2023)
31.10
0.037
0.096
2.00
4DGS (Wu al., 2023)4
31.02
0.030
0.150
36
4DGS (Ours)
32.01
0.014
0.055
114
initialize the Guassians for this dataset, we utilize the colored point cloud generated by COLMAP from the first frame of each scene. The timestamps of each point are uniformly distributed across scene's duration.
D-NeRF dataset (Pumarola et al., 2020) is monocular video dataset comprising eight videos of synthetic scenes. Notably, during each time step, only a single training image from a specific viewpoint is accessible. To assess model performance, we employ standard test views that originate from novel camera positions not encountered during the training process. These test views are taken within the time range covered by the training video. In this dataset, utilize 100,000 randomly selected points, evenly distributed within the cubic volume defined by [-1.2, 1.2]3, and set their initial mean the scene's time duration.
4.2 IMPLEMENTATION DETAILS
To assess the versatility of our approach, we did not extensively fine-tune the training schedule across different datasets. By default, we conducted training with a total of 30,000 iterations, batch size of 4, and halted densification at the midpoint of schedule. We adopted the settings of Kerbl et al. (2023) for hyperparameters such as loss weight, learning rate, and threshold. At the outset of training, we initialized both qi and gr as unit quaternions to establish identity rotations and set the initial time scaling to half of the scene's 's duration. While the 4D Gaussian theoretically extends infinitely, we applied a Gaussian filter with marginal p(t) < 0.05 when rendering the view further initialized the Gaussian with 100,000
time t. For scenes in the Plenoptic Video dataset, extra points distributed uniformly on the sphere encompassing the entire scene to fit the distant background that colmap failed to reconstruct and terminate its optimization after 10,000 iterations. Following the previous work, the LPIPS (Zhang et al., 2018) in the Plenoptic Video dataset and the D-NeRF dataset are computed using AlexNet (Krizhevsky et al., 2012) and VGGNet (Simonyan & Zisserman, 2014) respectively.
4.3 RESULTS OF DYNAMIC NOVEL VIEW SYNTHESIS
Results on the multi-view real scenes Table presents quantitative evaluation on the Plenoptic Video dataset. Our approach not only significantly surpasses previous methods in terms of rendering quality but also achieves substantial speed improvements. Notably, it stands out as the sole method capable of real-time rendering while delivering high-quality dynamic novel view synthesis within this benchmark. To complement this quantitative assessment, also offer qualitative comparisons on the "flame salmon" scene, as illustrated in Figure 3. The quality of synthesis in dynamic regions notably excels when compared other methods. Several intricate details, including the black bars

---

Published as a conference paper at ICLR 2024
Table 2: Qualitative comparison monocular dynamic scenes. The results are averaged over all scenes in the D-NeRF dataset. !: rendering at 800x800, otherwise downsampled 2x by default.
Method
PSNR
SSIM
LPIPS 4
D-NeRF (synthetic, monocular)
T-NeRF (Pumarola al., 2021)
29.51
0.95
0.08
D-NeRF (Pumarola et al., 2021)
29.67
0.95
0.07
Vox (Fang et al., 2022)
32.67
0.97
0.04
HexPlanes (Cao & Johnson, 2023)
31.04
0.97
0.04
K Plaaeee-explict (Fridovich-Keil et al., 2023)
31.05
0.97
K- X-Planes-hylln (Fridovich-Ko) et al., 2023)
31.61
0.97
V4D (Gan al., 2023)
33.72
0.98
0.02
4DGS (Wu al., 2023)
33.30
0.98
0.03
4DGS (Ours)
34.09
0.98
0.02
Ours (114 fps)
DyNeRF (0.015 fps) (Li al,, 20226)
Planes (0.23 fps)
NeRFPlayer (0.045 fps) (Song et al., 20233
HyperReel (2.00 fps) (Attal et al., 2023)
(Fridovich Keil et al., 2023)
Ground truth
Neural Volumes
LLFF
HexPlane (0.56 fps)
Mix Voxels (16.7 Tps) (Wang et 2023)
(Lombardi al., 2019)
(Mildenhall et 2019)
(Cao 2023)
Figure 3: Qualitative result on the flame salmon. It can be clearly seen that the visual quality is higher than other methods in the region from the moving hands and flame gun to the static salmon.
on the flame gun, the fine features of the right-hand fingers, and the texture of the salmon, are faithfully reconstructed, demonstrating the strength of our approach.
Results on the monocular synthetic videos We also evaluate our approach monocular dynamic scenes, a task known for its inherent complexities. Previous successful methods often rely archi- tectural priors handle the underlying topology, but we refrain from introducing such assumptions when applying our 4D Gaussian model to monocular videos. Remarkably, our method surpasses all competing methods, as illustrated in Table 2. This outcome underscores the ability of our 4D Gaussian model to efficiently exchange information across various time steps.
4.4 ABLATION AND ANALYSIS
Coherent comprehensive 4D Gaussian Our novel approach involves treating 4D Gaussian distri- butions without strict separation of temporal and spatial elements. Section 3.2, we discussed expressed in equation 6. This
an intuitive method to extend 3D Gaussians to 4D Gaussians, method assumes independence between the spatial (ar, y, z) and temporal variable I, resulting a block diagonal covariance matrix. The first three rows and columns of the covariance matrix can be processed similarly to 3D Gaussian splatting. We further additionally incorporate ID Gaussian to account for the time dimension.
To compare our unconstrained 4D Gaussian with this bascline, we conduct experiments on two representative scenes, shown in Table 3. We observe the clear superiority of our unconstrained 4D Gaussian over the constrained baseline. This underscores the significance of our unbiased and coherent treatment of both space and time aspects in dynamic scenes.
4D Gaussian is capable of capturing the underlying 3D movement Incorporation of 4D rotation our 4D Gaussian equips it with the ability to model the motion. Note that 4I) rotation can result in
8

---

Published as a conference paper at ICLR 2024
Table Ablation studies. We ablate our framework on two representative real scenes, flame salmon and cut roasted beef, which have volumetric effects, non-Lambertian surfaces, and different lighting conditions. "No-4DRot" denotes restricting the space and time independent of each other.
Flame Salmon
Cut Roasted Beef
Average
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
No-4DRot
28.78
0.95
32.81
0.971
30.79
0.96
No-4DSH
29.05
0.96
33.71
0.97
31.38
0.97
No-Time split
28.89
0.96
32.86
0.97
30.25
0.97
Full
29.38
0.96
33.85
0.98
31.62
0.97
Coffee Mariini
Cook Spinach
Cut
Salmon
Flame Steak
Render How
GT Flow
Render Image
GT Image
Figure 4: Visualization of the emerged dynamics of our 4D Gaussian The displayed views are selected from the test view of the Plenoptic Video dataset. The ground truth optical flows are extracted by VideoFlow (Shi et al., 2023) for reference.
a 3D displacement. To assess this scene motion capture ability, we conduct a thorough evaluation. For each Gaussian, we test the trajectory in space formed by the expectation of its conditional distribution Haysit- Then, we project its 3D displacement between consecutive two frames to the image plane and render it using equation 6 as the estimated optical flow. Figure 4, we select one frame from each scene in the Plenoptic Video dataset to exhibit the rendered optical flow. The result reveals that without explicit motion supervision or regularization, optimizing the rendering loss alone can lead to the emergence of coarse scene dynamics.
More ablations To examine whether modeling the spatiotemporal evolution of Gaussian's 's appear- ance is helpful, we ablate 4DSH in the second row of Table 3. Compared to the result of our default setting, we can find there is indeed decline the rendering quality. Moreover, when turning our attention to 4D spacetime, we realize that over-reconstruution may occur in more than just space. Thus, we allow the Gaussian to split in time by sampling new positions using complete 4D Gaussian PDF The last two rows in Table 3 verified the effectiveness of the densification in time.
CONCLUSION
We introduce a novel approach to represent dynamic scenes, aligning the rendering process with the imaging of such scenes. Our central idea involves redefining dynamic novel view synthesis by approximating the underlying spatio-temporal 4D volume dynamic scene using a collection of 4D Gaussians. Experimental results across diverse scenes highlight the remarkable superiority of our proposed representation, not only achieving state-of-the-aa rendering quality but also delivering substantial speed improvement over existing alternatives. To the best of our knowledge, this work stands as the first ever method capable of real-time, high-fidelity video synthesis for complex, real- world dynamic scenes.

---

Published as a conference paper at ICLR 2024
Acknowledgments This work supported in part by STI2030-Major Projects (Grant No. 202 2021200200244 National Natural Science Foundation of China (Grant No. 62106050 and 62376060), Natural Science Foundation of Shanghai (Grant No. 22ZR USyd-Fudan BISA Flagship Research Program and and Lingang Laboratory (Grant No. L.G-QS-20220--07)
REFERENCES
Jad Abou-Chakra, Feras Dayoub, and Niko Süinderhauf. Particlenerf: Particle based encoding for online neural radiance fields in dynamic scenes. arXiv preprint, 2022. 2,3
Benjamin Attal, Jia-Bin Huang, Christian Richardt, Michael Zollhoefer, Johannes Kopf, Matthew O'Toole, and Changil Kim. HyperReel: High-fidelity 6-DoF video with ray-conditioned sam- pling. CVPR, 2023. 7,
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin Brrtinn-ruuala,, and Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In ICCV, 2021. 1,2
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In CVPR, 2022. 2
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf:
Anti-aliased grid-based neural radiance fields. arXiv preprint, 2023. 2
Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. In CVPR, 2023. 1,
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In ECCV, 2022. 2
Anpei Chen, Zexiang Xu, Xinyue Wei, Siyu Tang, Hao Su, and Andreas Geiger. Factor fields: A unified framework for neural fields and beyond. arXiv preprint, 2023.
Guikun Chen and Wenguan Wang. A survey on 3d gaussian splatting. arXiv preprint. 2024. 3
Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia Conference Pupers, 2022.
Sara Kridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In CVPR, 2022. 2
Sara Fridovich-Keil Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023. 1, 7,8
Wanshui Gan, Hongbin Xu, Yi Huang, Shifeng Chen, and Naoto Yokoya. V4d: Voxel for 4d novel view synthesis. IEEE TVCG, 2023. 8
Tao Hu, Shu Liu, Yilun Chen, Tiancheng Shen, and Jiaya Jia. Efficientnerf efficient neural radiance fields. In CVPR, 2022. 2
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splat- ting for real-time radiance field rendering. ACM Transactions on Graphics (TOG), 2023. 3, 4, 5, 6,
Agelos Kratimenos, Jiahui Lei, and Kostas Daniliddis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiv preprint, 2023. 3
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012. 7
Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for 3d
video synthesis. In NeurIPS, 2022a.
10

---

Published as a conference paper at ICLR 2024
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In CVPR, 2022b. 6,
Yiqing Liang, Numair Khan, Zhengqin Li, Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint, 2023.
Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh. Neural volumes: learning dynamic renderable volumes from images. ACM Transactions on Graphics (TOG), 2019. 8
Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3d gaussians: Tracking by persistent dynamic view synthesis. In 3DV, 2024. 2,3
Ben Mildenhall, Pratul P Srinivasan, Rodrigo Ortiz-Cayon, Nima Khademi Kalantari, Ravi Ra- mamoorthi, Ren Ng, and Abhishek Kar. Local light field fusion: Practical view synthesis with prescriptive sampling guidelines. ACM Transactions Graphics (TOG), 2019. 7,
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020. 1,2
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics prim- itives with multiresolution hash encoding. ACM Transactions on Graphies (TOG), 2022. 2
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francese Moreno-Noguer. D-NeRF: Neural
Radiance Fields for Dynamic Scenes. In CVPR, 2020. 2, 3, 6,
Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer, D-nerf: Neural radiance fields for dynamic scenes. In CVPR, 2021. 8
Xiaoyu Shi, Zhaoyang Huang, Weikang Bian, Dasong Li, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, Jifeng Dai, and Hongsheng Li. Videoflow: Exploiting temporal cues for multi- frame optical flow estimation. arXiv preprint, 2023.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint, 2014.
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and An- dreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE TVCG, 2023. 2, 7, 7,8
Cheng Sun, Min Sun, and Hwann-Tzowe Chen. Direct voxel grid optimization: Super-fast conver- gence for radiance fields reconstruction. In CVPR, 2022.
Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srini- vasan. Ref-nerf: Structured view-dependent appearance for neural radiance fields. In CVPR, 2022. 1,2
Feng Wang, Sinan Tan, Xinghang Li. Zeyue Tian, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In ICCV, 2023. 1,3, 8
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Wang Xinggang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint, 2023. 3, 8
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Seg- former: Simple and efficient design for semantic segmentation with transformers. In NeurIPS, 2021. 13
Ziyi Yang, Xinyu Gao. Wen Zhou, Shaohui Jiao, Yuging Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint, 2023. 3

---

Published as a conference paper at ICLR 2024
Kai Zhang, Germot Riegler, Noah Snavely, and Vladlen Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint, 2020.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In CVPR, 2018.
Matthias Zwicker, Hanspeter Pfister, Jeroen Van Baar, and Markus Gross. Ewa splatting. /EEE TVCG, 2002.
A APPENDIX
A LIMITATIONS
While the first row of Figure 5 reveals that the 4D Gaussian can adeptly recover subsequent time steps exhibiting sigmificant geometric deviations from the first frame via densification and optimiza- tion, using point clouds extracted from only the first frame, and achieves high fidelity in the fore- ground dynamic regions. Nevertheless, in the absence of initial points, our approach is difficult to capture distant background areas, even they are static. Although some techniques such spherical initialization that we employed can mitigate this to an extent (for comparison, we turn off spherical initialization in the scene coffee martini Figure 5, where is evident that the exterior background was not successfully synthesized compared to the scene flame salmon), it does not suggest that get the correct geometry only a background map represented by a sphere of Gaussians is learned. These issues might constrain the convenience of our method in some scenes.
B PROOFS
In this chapter, will prove the properties which the main text relies when treating the unnor- malized Gaussian defined Eq. equation as a special probability distribution.
First we will prove Gaussian conditional probability formula also holds in equation and equation 6, iee
v( p(t)p(u, vlt).
(12)
where
p(u, v, 1) = (2x) det (Σ) N (u,
(13)
p(t) (2x) det (E.) & NNtllu Ex)
(14)
p(u, v/6) (2+)-1 det (Euvlt) 3/2 N Mlpulles Envje)-
(15)
To prove equation 12, since Gaussian conditional probability formula holds for normalized version: N( v, t|u, Σ) (t (,)((u, vlp it, Emnjt), only need to prove
det (E) = det (Ee) det (Eunjt).
(16)
From Gaussian property, we know that Emoje Ene Env, En - Then from the decomposi- tion of > by
Env Duvit
Exeat Σt
Σ
(17)
Σ Exe, Ex
Eyp
Euv,e Ei,av
[ 1]
0
Σg
- E Eraw
(18)
equation 16 holds immediately. p(0)(x, y, =(t).
The proof can be easily extended onto p(z,8,2,t)
=
12

---

Published as a conference paper at ICLR 2024
C ADDITIONAL QUANTITATIVE RESULTS AND VISUALIZATIONS
In Table 4, provide the PSNR breakdown different scenes. Figure 5 shows synthesis results at different timesteps for each scene in the Plenoptic dataset. The quanlitative results clearly show that 4DGS capable faithfully capture the subtle movement of cookware. From the result in Cut Roasted Beef, we can find that even though we only use the point cloud extracted from the first frame as the initialization of the Gaussians, still able to fit the body after large movement with high fidelity.
Coffee Martini
Spinach
Cut Beef
Flame Salmon
Flame Steak
Sear Steak
Mean
HexPlane
32.04
32,55
29.47
32.08
32.39
31.70
K exPlaccs-explicit
28.74
32.19
31.93
28.71
31.80
31.89
30.88
K Planes-hybrid
29.99
32.60
31.82
30.44
32.38
32.52
31.63
Mix Voxels
29.36
31.61
31.30
29.92
31.21
31.43
30.80
NeRFPlayer
31.53
30.56
29.353
31.65
31.93
29.12
30.69
HyperReel
28.37
32.30
32.922
28.26
32.20
32.57
31.10
Ours
28.33
32.93
33.85
29.38
34.03
33.51
32.01
Table 4: Per-scenes results on the Plenoptic Video dataset. we colored the Best, Second and Third results in each scene.
D RESULTS IN THE URBAN SCENES
Urban street bustling with numerous moving vehicles and pedestrians one of the most common dynamic scenes in daily life. Reconstruction of such scenes has great value as it can provide rich data for the training of autonomous driving models and be used for offline perception.
To test the applicability of the proposed 4D Gaussian the urban scenes, we selected several segments containing dynamic objects from the widely used Waymo Open Dataset. Each segment contains sequence of calibrated images captured by five pinhole cameras and LiDAR point clouds which can not only provide an accurate initialization 4D Gaussians but also be used for depth supervision. Following the previous work, we use images captured from three frontal cameras.
While the motion in urban scenes tends to less intricate than that in typical indoor dynamic novel synthesis datasets, the sparse observation and the wide range pose different challenges. To mitigate the potential overfitting, we integrate sparse depth supervision sourced from LiDAR point clouds, given by the (inverse) L1 loss and deactivate the temporal coefficient of 4DSH. Besides, adopt a cube map as the background model to model the sky with infinite distance and penalize the inverse depth in the sky area. The sky mask obtained using SegFormer (Xie al., 2021).
In Figure 6, we provide the qualitative result of reconstruction. As can be seen, 4D Gaussian Splat- ting achieves high-fidelity rendering for both dynamic and static regions. More video results can be found in the supplementary material.
Furthermore, we present the novel view synthesis results in Figure Following the common prac- tice, we take out one frame from every ten frames as the test view. Unlike the previous approaches typically rely the 3D bounding boxes and dynamic object segmentations, provide unified representation of both dynamic and static regions with the aforementioned modification.
E THE TEMPORAL CHARACTERISTIC OF 4D GAUSSIANS
If the 4D Gaussian has only local support in time, the 3D Gaussian does in space, the num- ber of 4D Gaussians may become very intractable as the video length increases. Fortunately, the anisotropic characteristic of Gaussian offers prospect of avoiding this predicament. To further un- leash the potential of this characteristic, we set the initial time scaling to half of the scene's 's duration mentioned in Section 4.2.
13

---

Published as a conference paper at ICLR 2024
Cut beef
Flame Salmon
Scar steak
Cook spinach
Coffee martini
Flame steak
Figure 5: View synthesis results different times.
order to more intuitively comprehend temporal distribution of the fitted 4D Gaussian, Fig- ure 8 presents visualization of mean and variance in the time dimension, by which the marginal distribution on of 4D Gaussians can be completely described.
It can be observed that these statistics naturally form mask to delincate dynamic and static regions, where the background Gaussians have a large variance in the time dimension, which means they are long time period. Actually, shown in Figure 9, the background Gaussians able
able to to be active throughout the entire time span the scene, which allows the total number of Gaussians to grow very restrained with the video length extending.
Moreover, considering that we filter the Gaussians according to the marginal probability (t(t) at negligible time cost before the frustum culling, the number of Gaussians actually participated in the
14

---

Published as a conference paper at ICLR 2024
S 0000777000 20
OPI Lrop
30
78077108 Se801
Figure 6: Visualization of urban scene reconstructions. rendered depth maps. Best viewed with zoom-in.
Left: rendered RGB images. Right:
rendering of each frame is nearly constant, and thus the rendering speed tends to remain stable with the increase of the video length. This locality instead makes approach friendly to long videos in terms of rendering speed.
In Figure 10, directly show the total number of Gaussians and the number of Gaussians really involved in the rasterization for a given frame under different video lengths. As it can seen, the total number of 4D Gaussians fitted on the video with hundreds of frames is not essentially larger
15

---

Published as a conference paper at ICLR 2024
PSNR: 30.17
PSNR: 27.55
PSNR:: 30.38
Figure 7: Novel view synthesis in the urban scenes.
Ref Image
Mean
Variance
Figure 8: The temporal distribution the fitted 4D Gaussian on the cut roasted beef. For better visualization, show the distance between the mean and the rendered timestamp.
than that of 3D Gaussians fitted on single frame and the average number of 4D Gaussians really used in rendering each frame stable.
We compare the sliced 3D Gaussians of two variants (No-4DRot and Full) in Figure 11. It can be obviously observed that under the No-4DRot setting the rim of the wheel is not well reconstructed. and fewer Gaussians are engaged in rendering the displayed frames after filter, despite a larger total number of fitted Gaussians under this configuration. This indicates that the 4D Gaussian in the No-4DRot setting has less temporal variance, which impairs the capacity of motion fitting and exchanging information between successive frames, and brings more flicker and blur the rendered videos.
16

---

Published as a conference paper at ICLR 2024
Figure 9: The marginal distribution the fitted 4D) Gaussian in time dimension.
25
total
Gaussians
number of render Gaussians (millios)
number of total Gaussions (millions)
2.0
2
1.5
G
1.0
to
0.6
50
video length (number frames)
Figure 10: The number of Gaussian at 10k iterations under different video lengths. The data at 1 frame denotes the number of 3D Gaussians fitted on the initial frame.
17

---

Published as a conference paper at ICLR 2024
Figure 11: Visualization of the time slices under Full setting (top) No-4D Rot (bottom). In the first column, we show the time slices of fitted 4D Gaussian under different settings. In the other columns, present the rendered images.
18


Reviews:
[{"review_id": "wmD2iRvFAl", "reviewer": "Reviewer_XBJp", "summary": "This submission introduces a 4D spatial-temporal representation (3D for space and 1D for time) based on Gaussian principles to model dynamic 3D scenes. This representation is a principled extension of the 3D Gaussian representation used in Gaussian Splatting for static scenes. With the 4D Gaussian representation, the states of 3D Gaussians at specific time instances become slices of the 4D function, enabling their rasterization into video frames for forward image rendering and scene reconstruction. The submission demonstrates that by modeling the interplay between space and time, implemented as a non-block-diagonal 4D covariance matrix, we can improve the modeling and reconstruction of dynamic scenes. The paper illustrates that with only the reconstruction loss and no additional regularizations, the 4D Gaussian-based representation achieves superior performance compared to state-of-the-art methods.", "questions": "1. The number of 4D Gaussians might be huge (millions in 3D GS paper), given that one more temporal dimension in addition to 3D space as in 3D GS. Can the author address that, since that might be one limitation of the Gaussian based representation?\n\n2. Are the 4D means of the Gaussians also being optimized? If so, the 4D means can also introduce displacements to 3D Gaussians. In other words, both 4D mean offsets, and 4D rotation can lead to displacements. Will this introduce ambiguity to explain the scene flow field?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+The proposed 4D Gaussian based representation to model the spatial-temporal field is a very principled extension of the 3D GS method. As a result, the fitting process is robust - no need to introduce additional regularization terms or optimization strategy to make sure the optimization will converge.\n\n+The scene reconstruction using the representation achieves SOTA performance with only reconstruction loss across all experiments.  No manually designed regularization term is needed. This shows the effectiveness of the 4D representation. \n\n+The 4D densification and pruning operations in spacetime can be derived naturally by extending from 3D. As shown in the submission, those operations improve the performance on modeling scene motions\n\n+The paper is well written with consistent and accurate notations, thus easy to follow and grasp.", "weaknesses": "- more discussion needed for key advantage of the proposed model:\nAlthough in Sec 3.2 and Tab.3, the authors mentioned the benefit of making space and time dependent by allowing non-block diagonal 4D covariance matrix for modeling motions,  it is not easily seen why the performance increase (full version vs \"No-4DRot\"). The visualization of temporal slices of the fitted 4D Gaussian could be helpful to demonstrate how the corresponding 3D Gaussian across frames models motions\n\n- more discussion on design choice needed:\nThe 4D SH function is also a function of time t, will it also explain the temporal appearance change (eg. motion)? if so, it is explaining the properties we did not intend it to (we use it for view-dependency), and it may interact with other optimized variables such as 4D rotation to explain the motion, will it lead to ambiguity and affect the performance of dynamic modeling ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sDyBhWkRRQ", "reviewer": "Reviewer_1bTN", "summary": "This paper generalizes the 3D Gaussian splitting to 4D. The key contribution is a 4D Gaussian-based representation for dynamic rendering. Unlike intuitively modeling the 4D as canonical 3D + deformations, this paper inserts many 4D bubbles into 4D space-time and jointly assembles the dynamic scenes. \n\nInterestingly, when using full rotational 4D Gaussians, to render a specific time, one just needs to slice the 4D distribution and then do standard 3D Gaussian EWA. The skew in 4D also leads to time-dependent center changing, which leads to local scene flow.\n\nThe proposed representation is tested with the dynamic rendering task on D-NeRF and DyNeRF datasets and shown to be effective. With the power of splitting, the proposed representation achieves high inference FPS.", "questions": "Pelase see weakness", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- It’s interesting to use local 4D Gaussians to represent full 4D functions instead of using explicit flow plus 3D Gaussian.\n- The introduction of full 4D covariance and the usage of the time-dependent harmonics of appearance is effective.\n- The flow extracted from the time-dependent center sliced from 4D Gaussians makes this 4D function approximation more reasonable because it may locally capture correspondence, which introduces the smoothness into the 4D approximation.", "weaknesses": "- One key weakness the reviewer is curious about is the number of Gaussians in the proposed representation when the video grows longer. And how long does each 4D Gaussian cover in time? If the 4D Gaussians, like the 3D static one, only have local support in time, then the memory or storage consumption may be huge, preventing applying such representation to long videos or streaming.\n- To help the readers understand the 4D Gaussians, one 2D/1D slice + time visualization may be helpful.\n- The LPIPS is not reported in the table while most dynamic rendering does.\n- Although the local flows as I write in the strengths are interesting, the reviewer is wondering how can these local correspondences generalize to global and long-term ones. It’s unclear whether such 4D mixture models can capture long-term tracking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "axhaPXESGy", "reviewer": "Reviewer_8jaG", "summary": "This work is on dynamic scene representation and rendering.\nThe authors extend Gaussian Splatting to dynamic scenes by extending the 3D Gaussian formulation with $t$, along with 4D spherindrical harmonics.\nExperiments are conducted on Plenoptic Video and D-NeRF dataset and demonstrate the effectiveness of the proposed method: more realistic and faster.\nAnalysis shows that some primitive underlying 3D movement can be captured.", "questions": "I know these are concurrent work (one even come after ICLR submission deadline). It would be admired if you can add in the final version about comparison to the following two works on extending Gaussian splatting for dynamic scenes.  I have no conflict of interests to these works.\n1. 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering. https://arxiv.org/pdf/2310.08528.pdf\n2. Dynamic 3D Gaussians: Tracking by Persistent Dynamic View Synthesis  https://arxiv.org/pdf/2308.09713.pdf\n\nBesides, it would be appreciated if you can apply the proposed method on any urban scenes (e.g. KITTI, Waymo dataset). These are the more challenging and common dynamic scenes in real world.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The extension of 3D Gaussian for dynamic scenes. It allows using 3D Gaussian to render dynamic scenes realistically.\n2. Overall the paper is clearly written and well explained in the motivation, design and implementation.\n3. Superior results compared to very recent SOTA methods include K-Planes, HexPlanes. Higher realism and speed.", "weaknesses": "* Concerns Regarding Novelty:\n   - The enhanced speed and realism seem to be primarily attributed to the power of Gaussian splatting, rather than the novel modules introduced.\n   - The implementation of 4D spherical harmonics appears to offer only a marginal improvement, as evidenced by the mere +0.2 PSNR increase in Table 3.\n   - The inclusion of time (t) in addition to spatial dimensions $(x,y,z)$ in the Gaussian model appears to be a straightforward extension, akin to extending TensoRF’s triplanes to K-planes.\n* Modeling Concerns: \n   - I have reservations about the authors’ approach of modeling the 4D Gaussian by treating space (xyz) and time (t) equally. This seems to suggest that each Gaussian fades in, peaked at $\\mu_t$ and out, as indicated by the term $p$ in Equations 4 and 10. This is not grounded in physical reality, where Gaussians are expected to move through space rather than appearing and disappearing abruptly.\n  - Consequently, I think this extension mainly increases the model’s capacity to incorporate time and I question whether the proposed approach can yield consistent results in more complex scenarios (e.g., higher moving speeds, sparse observations).\n\n\nThat being said, I think there are more effective ways to extend 3D Gaussian splatting for dynamics with a more accurate and physically-grounded representation. Nevertheless, I still consider this work to be above the bar in this field, providing some insights for the community.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)