Swift4D: Adaptive divide-and-conquer Gaussian Splatting for compact and efficient reconstruction of dynamic scene
Authors: Jiahao Wu
Rui Peng
Zhiyan Wang
Lu Xiao
Luyang Tang
Jinbo Yan
Kaiqiang Xiong
Ronggang Wang
Venue: ICLR 2025 Poster (2025)
Keywords: 3D Gaussian Splatting
Dynamic scene reconstruction.
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We realize the reconstruction of dynamic scenes through dynamic and static decomposition, which far exceeds the current sota method in terms of training and inference speed, rendering quality, and model size.

Abstract: Novel view synthesis has long been a practical but challenging task, although the introduction of numerous methods to solve this problem, even combining advanced representations like 3D Gaussian Splatting, they still struggle to recover high-quality results and often consume too much storage memory and training time. 
In this paper we propose Swift4D, a divide-and-conquer 3D Gaussian Splatting method that can handle static and dynamic primitives separately, achieving a good trade-off between rendering quality and efficiency, motivated by the fact that most of the scene is the static primitive and does not require additional dynamic properties. Concretely, we focus on modeling dynamic transformations only for the dynamic primitives which benefits both efficiency and quality. We first employ a learnable decomposition strategy to separate the primitives, which relies on an additional parameter to classify primitives as static or dynamic. For the dynamic primitives, we employ a compact multi-resolution 4D Hash mapper to transform these primitives from canonical space into deformation space at each timestamp, and then mix the static and dynamic primitives to produce the final output. This divide-and-conquer method facilitates efficient training and reduces storage redundancy. Our method not only achieves state-of-the-art rendering quality while being 20× faster in training than previous SOTA methods with a minimum storage requirement of only 30MB on real-world datasets.

Published as a conference paper at ICLR 2025
SWIFT4D: ADAPTIVE DIVIDE-AND-CONQURR
GAUSSIAN SPLATTING FOR COMPACT AND EFFICIENT RECONSTRUCTION OF DYNAMIC SCENE
Jiahao Wu, Rui Peng, Zhiyan Wang, Lu Xiao,
Luyang Tang, Jinbo Yan, Kaiqiang Xiong. Ronggang Wang" Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology Shenzhen Graduate School, Peking University 2.30 2.012122 2022212200sse stu.. pku. edu. cn,
rgwang@pkusz edd. cn
275:22
TINR
FF8:114 Rogl 40gs
FR84100 "22.0
Osedital
Keptares
XL.6
21.4
4005
F15. 30
XL
FPS Micconel
The size cirtle sepreseuss of model
a75.0 ie Notiitaye
97503
228
2014
Timethours)
Figure 1: Our method demonstrates high-quality rendering, rapid convergence, and compact storage characteristics. It can achieve competitive result with just minutes of training. Additionally, with increased training iterations, our method excels handling finer details.
ABSTRACT
Novel view synthesis has long been a practical but challenging task, although the introduction of numerous methods to solve this problem, even combining advanced representations like 3D Gaussian Splatting, they still struggle to recover high-quality results and often consume too much storage memory and training time. In this paper we propose Swift4D, a alvide-and-conquer 3D Gaussian Splatting method that can handle static and dynamic primitives separately, achieving a good trade-off between rendering quality and efficiency, motivated by the fact that most of the scene is the static primitive and does require additional dynamic properties. Concretely, we focus on modeling dynamic transformations only for the dynamic primitives which benefits both efficiency and quality. We first employ learnable decomposition strategy to separate the primitives, which relies on an additional parameter to classify primitives as static or dynamic. For the dynamic primitives, to transform these primitives from canonical space into deformation space at each timestamp, and then mix the static and dynamic primitives to produce the final output. This divide-and-on andde--ed-conquer method facilitates efficient training and reduces storage redundancy. Our method not only achieves state-of-the-arr rendering quality while being 20x faster training than previous SOTA methods with a minimum storage requirement of only 30MB real-world datasets.
employ compact multi-resolution 4D Hash mapper
INTRODUCTION
Novel view synthesis (NVS) is a crucial task in computer vision and graphics, with significant applications in areas such as augmented reality (AR), virtual reality (VR), and content production.
"Corresponding author

---

Published as a conference paper at ICLR 2025
Deformed point
Canonical point
Static point
Dynamic point
Deformation path
(a) Dynamic NeRF method
(b) Dynamic 3DGS method
(c) Our decomposition method
Figure 2: Illustration of different dynamic scene rendering methods. (a) Pumarola et al. (202)
Park et al. (2021) proposes mapping deformation field points to canonical space, a widely adopted practice NeRF-based methods; (b) Wu al. (2024); Yang al. (2024) propose mapping canonical space points to the deformation field; (c) We propose dividing the points in canonical space into dynamic and static, and then mapping only the dynamic points to the deformation space.
The goal of NVS is to render photorealistic images from arbitrary viewpoints using 2D images video inputs. While recent advancements have achieved considerable success in static scenes, this task becomes particularly challenging when applied to dynamic scenes, where complexities introduced by object motion and temporal changes make accurate rendering significantly difficult.
Current NVS techniques can be broadly classified into two predominant approaches: neural rendering methods, exemplified by Neural Radiance Fields (NeRF) Mildenhall
(202I). and point cloud-based rendering techniques, such as 3D Gaussian Splatting Kerbl et al. (2023). 33. NeRFs have recently made significant strides achieving photorealistic rendering of static scenes, with subsequent works |Barron et al. (2021) 2022, 2023): |Riiee et al. (2023) further enhancing both quality and speed. Despite these advancements static scene rendering, NeRFs face significant challenges when extended to dynamic scenes, primarily due to the substantial training time and storage requirements. To overcome these obstacles, various approaches have been proposed. As shown in Fig 2 a). [). Pumaarla et al. I202I) and Park et al. (2021) leverage deformation fields to map deformation space at arbitrary timestamps to canonical space, effectively capturing dynamic scene changes. [i et al. (2021) and Gao al. (202T) employ scene flow model the motion trajectories within dynamic environments. [Ca & Johnson (2023) and anddovicchh Keil et al. (2023). decompose the 4D spacetime domain into multiple compact planes, thereby improving training and rendering speeds. Although these methods have achieved some degree of success, achieving high-quality real-time rendering remains challenging.
Compared to NeRF, 3DGS offers significant advantages, including real-time rendering and substantially reduced training time. Within the scope of dynamic modeling, several notable methods have emerged. As shown in (b). 4DGS Wu et al. (2024). inspired by HexPlane, introduces neural voxel encoder to model deformation relationships over time. 3DGStream [Sun al. (2024) utilizes a compact Neural Transformation Cache (NTC) to efficiently model the translation and rotation 3D Gaussians between two adjacent frames. RTGS Yang et al. (2023) treats spacetime as an integrated whole by optimizing set of 4D primitives, parameterized anisotropic ellipses that capture both geometry and appearance. Additionally, STGS Li et al. (2024) enhances standard 3D Gaussians with temporal opacity and motion/rotation parameters, effectively capturing both static and dynamic elements to model dynamic deformation.
While these approaches achieve higher-quality results with faster rendering times, they still face challenges related to long training time and heavy storage requirements. One potential limitation in their approach is the uniform treatment of all Gaussian points during the modeling process. However, we observe that static points, such as those in background regions, constitute the majority of the scene. These points exhibit minimal dynamic modeling. It is more efficient to partition the scene into static and dynamic points and model each separately. This strategy has the potential to significantly reduce computational overhead and storage requirements. Moreover, as demonstrated by Wang et al.) (202)), applying the same modeling technique to both dynamic and static points can cause blurring in dynamic regions due to the influence of static areas, ultimately compromising rendering quality.
deformation and therefore do require complex
In this paper, we introduce Swift4D, a method that simultaneously achieves fast convergence, compact storage, and real-time high-quality rendering. Our approach starts by decomposing Gaussian points into dynamic and static groups based on 2D multi-view images, incorporating an additional parameter d for differentiation. For temporal modeling, we employ deformation field approach using a compact Gaussian points from canonical space to deformation space arbitrary timestamps. Notably, as
4DHash and MLPs as the deformer, which maps dynamic
2

---

Published as a conference paper at ICLR 2025
shown in Fig. 2(c), temporal modeling is applied exclusively to the dynamic points, while static points are treated as temporally invariant, significantly reducing computational demands. This reduction in the number of dynamic points enables the 4DHash to concentrate on deformation, leading to faster convergence and improved rendering quality. Finally, we combine the static and dynamic Gaussian points render the final output. This approach also addresses the issue of blurring caused by static elements interfering with the time-aware multi-resolution 4DHash.
Our method achieves SOTA performance in terms of training and rendering speed, storage efficiency, and rendering quality. Furthermore, our supplement videos basketball and 2) demonstrate that our approach remains effective even in scenarios involving large movements. We will release our code and pre-trained models upon acceptance. In summary, the key contributions of our work are:
1) We propose novel method for decomposing dynamic 3D scenes into dynamic and static components based on 2D images, effectively reducing computational complexity. This method can be seamlessly integrated into existing dynamic approaches plug-and-play module to enhance quality.
2) We introduce compact multi-resolution 4DHash, with a footprint as small as 8MB, to effectively model the quality and accelerates training but also ensures efficient and compact storage.
domain. This approach not only enhances rendering
3) Our method achieves state-of- of the-art performance in training and rendering speed, storage, and high-quality output.
2 RELATED WORK
Novel View Synthesis. In recent years, novel view synthesis has garnered significant attention, leading to numerous breakthroughs. NeRFMMildnhall et al. (2021) pioneered this domain by leveraging multi-layer perceptrons (MLPs) combined with volume rendering to model 3D radiance fields, enabling image rendering from arbitrary viewpoints. Subsequent works aimed to enhance efficiency and quality. Methods such as Tensorf Chen et al. (2022). DVGO [Sun et al. Plenoxe(Fridovich- (2022). Plenoxe [Fridovich-Ka Keil et al. (2022), and Plenoctree Yu al. (2021) adopt grid-based
representations for faster training and rendering. Instant NGP Müller et al. (2022) further accelerates this process with a hash encoder, significantly reducing computation time. Meanwhile, MipNeRF Barron et al. (2021) and MipNeRF360 Barron et al. (2022) propose integrated positional encoding (IPH) to model conical frustums, effectively mitigating aliasing issues. More recently, 3DGS Kerbl et al. 2023) introduced a novel point-based rendering paradigm for novel view synthesis, achieving real-time rendering with high quality. This has spurred additional advancements, including Mipsplatting Yu et al. (2024a) for anti-aliasing, 2DGS HHuang et al. (202a)) for improved mesh extraction, and ScaffoldGS Lu et al. (2024) for large-scale scene rendering.
Novel View Synthesis for dynamic scene. Li et al. 2202 ; Lin al. (2024): Kratimenos et al. (202)) attempt to directly model the trajectories of moving points across the scene, but they continue encounter challenges related to storage. [Pumarola al. [021]; Park et al. (202)) Wu et al. (202);; Yang et al.] (2024) try to build a consistent canonical space across each time step and then employ deformer, mainly MLP-based and Muti-plane-baas
map this canonical space to deformation spaces at each timestamp. Huang et al. (2024b) focuses on monocular dynamic inputs, leveraging sparse control points to reconstruct scene dynamics with exceptionally high FPS.
Lin et al. 2024) employs Fourier series and polynomial fitting to model the motion of Gaussian points, enabling dynamic reconstruction. K-planes Fridovich- Keil et al. (023) and Hexplane
(agllaneCCao> Cao & Johnson @chnson(2025) (203) employ explicit structural representation of the 6D light field rather than modeling underlying motions. Representing the deformer using MI.s or low-rank planes can reduce storage requirements, but it often results in slower training and limited capacity for capturing complex deformations.
Recently, He et al. (2024); Yan et al. attempt to separate dynamic and static Gaussian points to improve rendering quality and introduced external models to segment foreground and background areas. While these efforts have explored this direction, the resulting output quality remains suboptimal. [Liang et al. (2023) employs adaptive dynamic-static separation, which differs from explicit separation approach.
3

---

Published as a conference paper at ICLR 2025
Static
Init.
Decomp.
Density control
Dynamic GS
4D hash Bad
MLPs
Deform
dr
> ?
(...F.o..ddd
Mixed GS
Rendered IMG
Spetial- Temporal Structure Sec.3.3
Init.
Sparse points
Decomp.
Canonical. points
Density control
Floater
Train view
V
Test view
First Frame Images
Dynamic masks
Sec.3.2
Sec.3.4
Pipeline of our Swift4D. First, use the first frame images obtain well-initialized canonical point cloud. Then, we train the dynamic parameter d according to the method described in Sec 3.2 Based on d, the point cloud is divided into dynamic and static categories. Dynamic
Figure 3:
points undergo deformation using a spatio-temporal structure, as discussed in Sec 3.3 Finally, the deformed dynamic points are mixed with static points for rendering.
METHOD
Our main approach aims to achieve faster training speeds and higher quality rendering results through the decomposition of dynamic and static elements. Based this insight, we designed our pipeline, as illustrated in Fig. 3 In this section, we will provide detailed analysis of each module the pipeline. The preliminary concepts of 3D Gaussian Splatting are briefly introduced in Sec. 3.1 We initially train the canonical space Gaussians using the first-frame images and then optimize the dynamic parameter d of each Gaussian point based on the 2D dynamic-static pixel masks from different viewpoints, as discussed in Sec In the following stage, as outlined in Sec. (33 we freeze the training of dynamic parameter and proceed to jointly optimize the remaining parameter of the Gaussian points alongside the swift spatio-temporal structure. Furthermore, our pruning strategies are thoroughly described in Sec. 3.4 while Sec. 3.5 provides an in-depth discussion of the optimization process.
3.1 GAUSSIAN SPLATTING PRELIMINARY
3erb et al. 2023 uses 3D Gaussian points as its rendering primitives. These 3D Gaussian points have the following parameter: mean μ. covariance matrix Σ, opacity o and view-dependent color c. A 3D Gaussian point is mathematically defined as:
G(x) - -it
(1)
In the next rendering phase, the 3D mean μ is directly projected onto the plane as 2D mean 2² while the 3D covariance matrix is transformed into a 2D covariance matrix using the following formula: Σ' = (JWEWT JT) JT), where W and J denote the viewing transformation and the Jacobian of the affine approximation the perspective projection transformation, respectively. Finally, the color of each pixel can be calculated using the following formula:
i-1
1 II (y)() where as(x) = og exp (- 10---,?00 --- 112
C(x)
Σ (2)5(x) (x
Where N the number of Gaussian points that intersect with the pixel x E R In the actual implementation, the covariance matrix Σ is typically decomposed into rotation q and scaling The color e represented by a spherical harmonics (SH) function. Therefore, a Gaussian point can be represented G( q, c).

---

Published as a conference paper at ICLR 2025
Dynamic
Static
Rendering
Gradient Now
Docluston
Impat view
(a)
(b)
Figure 4: (a) Diagram illustrating dynamic parameter d optimization. Even when static points (blue) are occluded by dynamic points (orange) from View 1, they can still be correctly optimized from View 2 and 3. (b) shows the result of decomposition. From top left to bottom right, the order is GT mask, dynamic parameter rendered image, dynamic point and static point rendering results.
3.2 EFFICIENT DYNAMIC AND STATIC DECOMPOSITION
In this section, we introduce our dynamic-static decomposition method for eliminating redundant computations for static Gaussian points. The time-varying motion model is applied solely to the dynamic components, leaving the static elements unchanged. This approach leads to faster convergence and enhanced rendering quality. Specifically, we introduce learnable dynamic parameter (Initialized to 0.) within the Gaussian points to quantify the the dynamic level of each points. A higher d corresponds more pronounced motion, indicating that the point is likely dynamic. We first compute a 2D dynamic-static pixel mask D(x) from the training videos to distinguish dynamic and static pixels, as shown Eq. 3 which serves as the supervision signal.
D(x)= {
S(x) >= 7 where S(x)
T (cz,))
((z, t)
T Σ C(x,t)) (3)
where C(x, represents the pixel intensity of in th frame location R S(x) is the temporal standard deviation (std) for each pixel across the entire time duration T'. Subsequently, a threshold of y 0.02 is applied binarize S(z), generating a pixel mask (xx). Pixels with S(x) greater than or equal to are classified as dynamic, while those below are considered static.
Based on the concept: During backpropagation, Gaussian points intersecting dynamic pixels should receive a positive gradient, while those intersecting static pixels should receive a negative gradient, with the gradient gradually weakening with distance and occlusion. Our decomposition design illustrated in Fig. 4a). We use the a composition for parameter d with Sigmoid function to render dynamic value D(x) at location as Eq. shows.
D(z) = Sigmoid Σ (a)(() (1
(4)
5=1
By applying the sigmoid function, we optimize the dynamic parameter d to span too), enabling finer differentiation of dynamic degrees. This approach converts our dynamic-static decomposition into a binary classification problem. Consequently, optimizing the dynamic value for each Gaussian point can be accomplished by minimizing the binary cross-entropy loss:
(5) From the equation above, effecttivly optimize the dynamic parameter d for the Gaussian points. The entire optimization process highly efficient, typically concluding within minute. Ultimately, Gaussian points with dynamic parameter greater than the dynamic threshold ( 7.0 classified as dynamic; otherwise, they are classified as static. An ablation study on is presented in Fig. and Sec. 5] demonstrating that our decomposition method is robust the choice of Ç.
L E, ((())))))((()) (1 (oz) D(x))].
Notably, our method adapts well to occlusion. In Fig. ()), while dynamic pixels (orange) from View incorrectly assign positive dynamic values static Gaussian points (blue), other views like View
5

---

Published as a conference paper at ICLR 2025
2 and 3 assign larger negative values, ensuring correct classification. Further details are provided in the Supplementary material Sec. A.2
3.3 SPATIO-TEMPORAL STRUCTURE
We introduce our proposed efficient spatio- structure encoder, the 4D multi nusti-resolltion hash hid, and the deformation decoder MLPs, used to predict the deformation of each dynamic Gaussian.
4D Multi-resolution hash encoder. Inspired by INGP Müller et al. (2022). we propose utilizing a 4D multi- multi-resolution hash ha for encoding to effectively model the temporal information of dynamic Gaussians by normalizing the point cloud into the hash grid range. As described in INGP, voxel grid at each resolution is mapped to hash table that stores F F-dimensional learnable feature vectors. For a given 4D dynamic Gaussian (μ, t) E IR,, its hash encoding at resolution I, denoted as h4a(H, t; () E R is computed through linear interpolation of the feature vectors associated with corners of the surrounding grid. Consequently, its multi-resslution hash encoding features are as follows:
f [,4a(p, t; 0), (aa t; 1)...h4a((4, t; L 1)] E R.F..
(6)
where L denotes the number of resolution levels, typically set to 16. Following this, small MLP da combines all features to produce fa @a(fa). Using the 4D hash had as encoder offers several advantages: compactness, O(1) query complexity, and the multi-resolution approach effectively integrates global and local information.
However, while 4D Hash offers O(1) query complexity, its hashing characteristics make encoding the temporal information of entire scene both challenging and mooaaee-ntenssvg Fortunately, our proposed decomposition method focuses on encoding the temporal information of dynamic points only, reducing the need for a larger hashing space and simplifying the modeling of the scene's temporal domain. This approach enables us to retain the fast access speed of 4D Hash while minimizing storage requirements.
Multi-head Gaussian Deformation Decoder. Once all features of dynamic Gaussian points are encoded, we can compute any required variables using multi-head Gaussian deformation decoder MLPs = on Pes 2qs Po, Ssh
dy, ds, diz do, dsh = MLPssfa)
Here, du, ds, dq, do, dsh represent the deformation intensity the mean, scaling, rotation, opacity, and color of the Gaussian point at time t. Therefore, the deformed parameters of dynamic Gaussian G2 can be expressed as:
(u','',d'o 0', sh') (μ + du.r r + dr, q + dg, a + do, sh + dsh)
o', where (jL' o' sh') represent the parameters of the dynamic Gaussian at time For static Gaussian elements G., they are directly combined with the deformed dynamic Gaussian elements Ga render the final rendered image Tt.
3.4 DENSITY CONTROL
In the original 3DGS, the opacity of all points is regularly reduced, and Gaussian points with low transparency are clipped during the pruning stage. However, this approach not appropriate for our method it results in excessive coupling between the canonical space and the deformation space. Therefore, we eliminate the reset opacity operation. Inspired by previous works Niemeyer et al.) 2024); Deng et al. (2024); Fan et al. 2023), which focus on the compact representation of static scenes by pruning redundant Gaussians based on spatial attributes such as transparency and volume. We adopt novel approach to pruning floaters across canonical and deformation space: Temporal Importance Pruning, as shown in Fig. 3 This involves calculating the importance of each Gaussian point each training viewpoint at every timestamp. Gaussians with importance below certain threshold can be clipped, effectively reducing floater issues. For a Gaussian point gs, the importance w; calculated as follows:
Wi
max (as (xlt)
II (1 (x(1))
JEZ,EELEET
/=1

---

Published as a conference paper at ICLR 2025
Table 1:
Quality comparison on the N3DV dataset. The best and the second best results are denoted by red and blue. online method.
Method
PSNR
DSSIM
LPIPS 4
Time
Size(MB) 1
FPS 1
DyNeRF Li et al. 2022b)
29.58
0.020
0.099
1300.0 hours
30
0.02
NeRFPlayer Song et al. (2023
30.69
0.111
6.0 hours
5100
0.05
HexPlane Caò & Johnson 2023
31.70
0.014
0.075
12.0 hours
240
0.21
K- Plaaeee Fridovich- Keil et a. 2023
31.63
0.018
5.0 hours
300
0.15
4DGS Wu el al. 2024
31.02
0.030
0.150
50 mins
90
30
3DGStream Sun et al. 2024
31.67
60 mins
2340
215
SpaceTimmeS Li cl al
2024
32.05
0.014
0.044
10.0 hours
200
110
Real-Time4DGSS Yang et
2023
32.01
0.014
0.055
9.0 hours
1000
114
Swift-DLite
31.79
0.017
0.072
20 mins
30
128
Swift4D(OOrss)
32.23
0.014
0,043
25 mins
120
125
Table 2: Quantitative comparison the MeetRoom dataset. PSNR is averaged across all frames, while training time and storage requirements accumulate over the entire sequence. online method.
Method
PSNR
Time(hours).
Size(MB)
Plenoxe Fridovich- Keil et al.
2022
27.15
70
304500
MulGer
2222
28.10
5.5
14460
3DGSKerbl
2023
31.31
13
6330
StreamRF Li al. 2022a
26.72
0.85
2700
3DGStream Sun et 2024
30.79
0.6
1230
(wirt
32.05
0.3
40
Here, I represents the images from all training views, a; (z((t) is the value of (xx) at time t in Eq. T represents the set of query times. We prune Gaussians when their importance satisfies wy < 0.02. As illustrated in Fig- [ this method effectively eliminates artifacts that suspended in the air and were not captured by the training views. For the cloning and splitting of Gaussians, we adhere to the procedures of 3DGS, with the child Gaussians inheriting the dynamic properties of their parent Gaussians.
3.5 OPTIMIZATION PIPELINE
We start by initializing the SIM Schonberger & Frahm (0066) point cloud using the first frames, then train on the first-frame images for 5000 iterations establish well-defined canonical space. Next, training the dynamic attributes of each Gaussian point within the canonical space takes about minute, followed by training the our loss function remains simple, without additional terms:
structure. Consistent with the principles of 3DGS,
Lree (1 A1)41 + A
EXPERIMENT
In this section, we provide details of our implementation and datasets in Sec. respectively. A thorough analysis of the experimental results is presented in Sec. 4.3 while Sec. 4.4
covers the ablation experiments for our method. The results show that our approach achieves sota performance in terms of training speed, storage efficiency, and rendering quality.
4.1 IMPLEMENTATION DETAILS
We initialize with point clouds generated by Colmap, followed by constructing our canonical space using the first frames from all training viewpoints, trained for 5000 epochs. Next, we train the dynamic parameter d of Gaussian points using the Adam optimizer Kingma & Ba (2014) with learning rate of 0.05. This training spans 3000 epochs and completes in under minute. Finally, train apatio-temporal structure for approximately 14000 epochs, utilizing settings for the 4D Hash table similar to those in InstantNGP Müller et al. (2022). We use the Adam optimizer with an initial learning rate of 0.002, which exponentially decays to 0.00002 the course of training.

---

Published as a conference paper at ICLR 2025
(a) GT
(b) Ours
(c) 3DGStream
(d) 3DGS
Figure Qualitative result on the discussion.
(a) GT
(b) Ours
(c) RTGS
(d) STGS
Figure 6: Qualitative result on coffee martini and cut beef. It can be observed that our method achieves higher-quality modeling in both dynamic and static regions.
Lite refers lite-version model with hash table size conducted on an NVIDIA RTX 3090 GPU.
2 and \ 0. All experiments were
4.2 DATASET
The N3DV dataset Li et al. (2022h) is captured using multi-view system with 18-21 cameras, recording dynamic scenes at a resolution of 2704 x 2028 and 30 FPS. includes various complex scenarios such fire, reflections, and new objects. Following prior works Li et al. (202b); Cao & Johnson (2023); Wu et al. (2024); Yang et al. (2023): Li et al. (2024). we downsampled the videos by factor of two and used the same training and testing data splits as established by them.
The Meet Room dataset Li et al. 2022a) is captured using a multi-view system with 13 cameras, recording dynamic scenes resolution of 1280 720 and 30 FPS. Following prior works [Sun et al. (022)); Li et al. (2022a), we used 12 views for training and reserved I view for testing.
The Basketball court dataset VRU 2024) captured using multi-view system with 34 cameras, recording dynamic scenes at a resolution of 1920 1080 and 25 FPS. This dataset encompasses large scene with many complex situations, including bouncing, fast motion, occlusion, and transient objects, making it highly challenging.
4.3 EVALUATION
For the Meetroom and Basketball court dataset, we follow the processing approach from Sun et al. (2024), using COLMAP(Schonberre as the global pose. For the N3DV dataset, we adopt the processing approach from Wu et (2024). We evaluate the methods using three metrics across all 300 frames: 1) Average PSNR, DSSIM, and LPIPS Zhang et al. (2018) scores for the test views; 2) Total training time and FPS; 3) Model size.
& Frahm| (2066) to estimate the camera pose of the first frame
8

---

Published as a conference paper at ICLR 2025
(a) Lite
(b) Muti-plane encoder
(c) W/O decomp
(d) Full(Ours)
Figure 8: Some ablation experiments results on sear steak.
Table Quantitative results comparison for sear steak and flame steak includes average PSNR and time metrics over 300 frames of the test view.
Tab. and Fig. respectively present the quantitative and qualitative evaluations of various methods the N3DV video dataset. As shown in Tab. [ our approach not only significantly surpasses previous methods in rendering quality but also achieves speeds at least 20 times faster compared to methods achieving similar rendering quality Yang et al. (2023) Li et al. (2024). It can be seen in Fig. 6 that our method not only achieves higher-quality modeling of static regions, such as the plate in the bottom right corner and the background outside the window in the coffee martini, but also provides more detailed modeling of moving regions, such as the arms. As for the MeetRoom dataset results, shown in Fig. [ and Tab. 2 our method achieves state-of-the-art performance in rendering quality, training time, and storage efficiency. Particularly noteworthy is the storage efficiency, 3DGStream requires 30 times more storage compared to our approach. The results of training on the basketball court dataset are presented in Fig. 10] Appendix Tab. [ and the supplementary videos (basketball I and 2), showeasing complex dynamic scenes. Our decomposition technique effectively separates all athletes from the scene, illustrating the model's strong adaptability to occlusion, as discussed in Sec. 3.2
Method
PSNR 1
Time(mins) +
Lite Muti-planes W/o decomp.
33.31
33.48
28
32.68
35
Full
33.83
25
method's ability to handle highly
4.4 ABLATION AND ANALYSIS
Dynamic and static decomposition. To validate the effectiveness of our dynamic-static decomposition method,
33.6
conducted experiments on the sear steak and flame steak. As illustrated in Fig. (]c) and Tab. 3 treating all points as dynamic led to increased computation time, significantly reduced rendering quality, and introduced blurring in areas with large motion amplitudes.
33.4
0.16
33.2
0.18
0.12
32.8
32.6
0.00
32.4
0.08
There are three commonly used choices for encoder selection: an implicit MLP'Gao et al. (202I). multi-planes
Muti-plane encoder.
0.02
peroentage points PSNR
32.2
Cao & Johnson 2023), and hash table Müller et al. (2022). In this study, we delve into employing the multi-plane approach replace the 4D Hash as the encoder in our method. The subjective and objective experimental results, shown Fig. b) and Tab. 3 indicate that while it slightly lags behind the hash table in terms of rendering speed and quality, still outperforms methods that do not apply dynamic-static decomposition Wu cct al. (2024).
Figure 7: Distribution of dynamic points counts and PSNR at different thresholds.
Temporal importance pruning, As shown in Fig. 9 (a) and (b) exhibit severe artifacts. In contrast, images rendered with our pruning strategy, (c), appear much cleaner.
DISCUSSION
Incomplete decomposition of dynamic and static points. Although we employ the pixel level supervisor, it fails to fully decouple dynamic and static points. This can lead two issues: Gaussian

---

Published as a conference paper at ICLR 2025
(a) With opacity reset
(b) W/O opacity reset
(c) Ours
(d) GT
Figure 9: Importance Pruning Ablation Experiments: (a), (b), (c), and (d) show the rendered
results of the our model with opacity reset every 3000 iterations, without opacity reset, with our importance pruning method, and the ground truth, respectively.
(C)
Figure 10: Basketball court dataset experiment. (a) and (c) are dynamic point renderings, while (b) and (d) are GT. The black floaters are actually Gaussian points from the dynamic background.
points in textureless regions of small moving objects may be mistaken for static, while static objects may be identified as dynamic due to interference from nearby moving objects.
In the first situation, as shown in Fig. b),, certain textureless areas, like clothing and the table, are mistakenly identified static, despite being dynamic. In fact, this proves beneficial in Tab. 3 If the entire table were labeled as dynamic, the increased dynamic points would lower rendering quality (W/o decomp). By recognizing only the edges dynamic, where pixel changes significant, the method reduces the number of dynamic points and enhances rendering quality (Ours).
In the second situation, illustrated in Fig. 10 some Gaussians in static areas classified as dynamic due to the shadows or movements of the basketball players passing through these regions. Therefore, it reasonable identify these areas dynamic. Classifying these points static will impact the visual experience, as static points can hardly model dynamic areas.
The selection of the dynamic threshold. As shown in Fig. experiments with the "cook spinach" scene revealed that varying the dynamic threshold from to 9 did not significantly affect PSNR or the percentage of dynamic points, demonstrating the robustness of our method. To ensure consistency, we set the threshold 7.
The training time. Due to the large dataset (nearly 6000 images), we load images during training, which empirically wastes around 40% of the training time on disk 1/O. Eliminating this overhead could reduce training time to 10 minutes while maintaining high-quality 4D scene reconstruction.
CONCLUSION
In this paper, we introduce Swift4D, which achieves fast convergence, compact storage, and high-quality real-time rendering capabilities within the field of 4D reconstruction. The core innovation of our method lies in the introduction of dynamic-static decomposition technique, which can be applied to most existing dynamic scene reconstruction methods, enhaneing quality and accelerating convergence. Additionally, we introduce a 4D Hash encoder and multi-head tempool structure, allowing for faster and more efficient temporal modeling of dynamic points. Finally, to prevent severe coupling between the canonical and deformation fields, propose novel temporal pruning method that effectively removes floaters in the scene. Our proposed method delivers competitive results in just 5 minutes, and we hope it can offer new insights for applications struggling with training efficiency.
decoder
Limitation: Similar to previous work Sun et al. (024); Li et al. 2024). our mrthod focuses on multi-view scenes and currently does not support monocular datasets for dynamic scene reconstruction. Additionally, our method focuses on scene reconstruction and does not include human reconstruction Wu et al. (2020); Cheng et al. (2023).
10

---

Published as a conference paper at ICLR 2025
7 ACKNOWLEDGEMENTS
This work is financially supported by Guangdong Provincial Key Laboratory of Ultra High Definition Immersive Media Technology(Grant No. 2024B1212010006) National Natural Science Foundation of China U21B2012, Shenzhen Science and Technology Program-Shenzhen Cultivation of Excellent Scientific and Technological Innovation Talents project(Grant No. 0BC200000114144350000 this work is also financially supported for Outstanding Talents Training Fund in Shenzhen.
REFERENCES
Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Pratul P Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 5855-5864, 2021.
and
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5470-5479, 2022.
Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pp. 19697-19705, 2023.
Ang Cao and Justin Johnson. Hexplane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 130-141, 2023.
Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European conference on computer vision, pp. 333-350. Springer, 2022.
Wei Cheng, Ruixiang Chen, Siming Fan, Wanqi Yin, Keyu Chen, Zhongang Cai, Jingho Wang. Yang Gao, Zhengming Yu, Zhengyu Lin, et al. Dna-rendering: A diverse neural actor repository for high-fidelity human-centric rendering. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 19982-19993, 2023.
Tianchen Deng, Yaohui Chen, Leyan Zhang, Jianfei Yang, Shenghai Yuan, Jiuming Liu, Danwei Wang, Hesheng Wang, and Weidong Chen. Compact 3d gaussian splatting for dense visual slam. arXiv preprini arXiv:2403.I1 1247, 2024.
Zhiwen Fan, Kevin Wang, Kairun Wen, Zehao Zhu, Dejia Xu, and Zhangyang Wang. Lightgaussian: Unbounded 3d gaussian compression with 15x reduction and 200+ fps. arXiv preprint vrXiv:2311.17245, 2023.
Sara Fridovich-Keil, Alex Yu, Matthew Tancik, Qinhong Chen, Benjamin Recht, and Angjoo Kanazawa. Plenoxels: Radiance fields without neural networks. In Proceedings the the JEEF/CVF conference on computer vision and pattern recognition, pp. 5501-5510, 2022.
Sara Fridovich-Keil Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12479-12488, 2023.
Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5712-5721, 2021.
Bing He, Yunuo Chen, Guo Lu, Li Song, and Wenjun Zhang. S4d: Streaming 4d real-world reconstruction with gaussians and 3d control points. arXiv preprint arXiv:2408. 13036, 2024.
Binbin Huang, Zehao Yu, Anpei Chen, Andreas Geiger, and Shenghua Gao. 2d gaussian splatting for geometrically accurate radiance fields. arXiv preprint arXiv:2403. 17888, 2024a.
11

---

Published as a conference paper at ICLR 2025
Yi-Hua Huang. Yang-Tian Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. So-gs: Sparse-controlled gaussian splatting for editable dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024b.
Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühhler and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Trans. Graph., 42(4))139--1 2023.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Agelos Kratimenos, Jiahui Lei, and Kostas Danillidis. Dynmf: Neural motion factorization for real-time dynamic view synthesis with 3d gaussian splatting. arXiv preprint arXiv:2312.00112, 2023.
Lingzhi Li, Zhen Shen, Zhongshu Wang, Li Shen, and Ping Tan. Streaming radiance fields for 3d video synthesis. Advances in Neural Information Processing Systems, 15:1555--33498, 2022a.
Tianye Li, Mira Slavcheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5521-5531, 2022b.
Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. Spacetime gaussian feature splatting for real-time dynamic view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8508-88888 -8088 2024.
Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-tima view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6498-6508, 2021.
Yiqing Liang, Numair Khan, Zhengqin Li, Thu lguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. Gaufre: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint arXiv:23/2./14:: 2023.
Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia 2022 Conference Papers, pp. 1-9, 2022.
Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-flow: 4d reconstruction with dynamic 3d gaussian particle. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21 1113362-1155 2024.
Tao Lu, Mulin Yu, Linning Xu, Yuanbo Xiangli, Limin Wang, Dahua Lin, and Bo Dai. Scaffold-gs: Structured 3d gaussians for view-adaptive rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20654-20664 2024.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM,
106, 2021.
Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding. ACM transactions graphics (TOG), 41(4): 1-15, 2022.
Michael Niemeyer, Fabian Manhardt, Marie-Julie Rakotosaona, Michael Oechsle, Daniel Duckworth, Rama Gosula, Keisuke Tateno, John Bates, Dominik Kaeser, and Federico Tombari. Radsplat: Radiance field-informet gaussian splatting for robust real-time rendering with 900+ fps. arXiv preprint arXivv2403. 13806, 2024.
Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla, Nerfies: Deformable neural radiance fields. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pp. 5865-5874, 2021.
12

---

Published as a conference paper at ICLR 2025
Albert Pumarola, Enric Corona, Gerard Pons-Moll. and Francese Moreno-Noguet D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10318-10327, 2021.
Christian Reiser, Rick Szeliski, Dor Verbin, Pratul Srinivasan, Ben Mildenhall, Andreas Geiger, Jon Barron, and Peter Hedman. Merf: Memory-eftie radiance fields for real-time view synthesis in unbounded scenes. ACM Transactions on Graphics (TOG), 42(4))1-12, 2023.
Johannes L Schonberger and Jan-Michael Frahm. Structure-from-mottin revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4104 1104-4113, 2016.
Liangchen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732-2742, 2023.
Cheng Sun, Min Sun, and Hwann-Tzong Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5459-5469, 2022.
Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3dgstream: On-the-fly training of 3d gaussians for efficient streaming of photo-realistic free-viewpoint videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 20675-20685,, 2024.
VRU.
2024.
Feng Wang. Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 19706-19716, 2023.
Qianqian Wang, Zhicheng Wang. Kyle Genova, Pratul P Srinivasan, Howard Zhou, Jonathan T Barron, Ricardo Martin-Brualle Noah Snavely, and Thomas Funkhouser. Ibrnet: Learning multi-view image-based rendering, In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp.
2021.
Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pp. 20310-20320, 2024.
Minye Wu, Yuehao Wang, Qiang Hu, and Jingyi Yu. Multi-view neural human rendering. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp- 1682-1691, 2020.
Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou. 4k4d: Real-time 4d view synthesis at 4k resolution. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Jinbo Yan, Rui Peng, Luyang Tang, and Ronggang Wang. 4d gaussian splatting with scale-aware residual field and adaptive optimization for real-time rendering of temporally complex dynamic scenes. In ACM Multimedia 2024.
Zeyu Yang. Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642. 2023.
Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp.
2024.
Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, and Angjoo Kanazawa. Plenoctrees for real-time rendering neural radiance fields. In Proceedings of the IEEE/CVF International Conference Computer Vision pp. 5752-5761, 2021.
13

---

Published as a conference paper at ICLR 2025
Zehao Yu, Anpei Chen, Binbin Huang, Torsten Sattler, and Andreas Geiger. Mip-splatting: Alias-free 3d gaussian splatting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19447-19456, 2024a.
Zehao Yu, Torsten Sattler, and Andreas Geiger. Gaussian opacity fields: Efficient and compact surface reconstruction in unbounded scenes, arXiv preprint arXiv:2404. 140,:2220..1772,, 2024b.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
Zheng Zhang, Wenbo Hu, Yixing Lao, Tong He, and Hengshuang Zhao. Pixel-gs: Density control with pixel-aware gradient for 3d gaussian splatting. arXiv preprint arXiv:2403. 15530, 2024.
A
APPENDIX
In the supplementary materials, will provide more details. In A.I we provide detailed setting s about our method. In Sec[A.2] we describe our dynamic-static decomposition method in detail. In we present additional experimental results.
Sec A.3
A.1
IMPLEMENT DETAILS
Three-stage method. In the first stage, we train the Gaussian points initialized by StMSchonberger & Frahm (2016) using the first frame images from each viewpoint. The goal this stage is to obtain a canonical space. In the second stage, we train the dynamic parameter d of each Gaussian point according to the method proposed in Sec. 3.2 In the third stage, we jointly train the attributes of the Gaussian points and the spatio-temporal structure.
we use five shallow MLPs as decoders for the mean, opacity, color, rotation, and scaling, respectively. The outputs directly added to the attributes the Gaussian points in the canonical space, and then passed through the corresponding activation functions to obtain the attributes time
MLPs as feature decoders. As shown in Fig 13
Lite version. We empirically found that removing the SSIM loss, while slightly degrading rendering quality, offers the advantage of reducing the number of Gaussian points by 2-3 times (approximately 200,000). Therefore, we removed the SSIM loss and set the hash table size to 215 in the I Lite version. This ensures a significant reduction in model size without severely impacting rendering quality. The models rendered in the Lite version average only 30MB in size, with the Gaussian point cloud being 22MB and the hash table 8MB, making the smallest dynamic model date (including the NeRF series).
A.2 DYNAMIC STATIC DECOMPOSITION
We precompute the temporal variance (x() for each pixel x E R to generate the variance map V for each viewpoint i. To reduce noise, smooth Vú using Gaussian filter with 31x31 kernel. Each pixel's variance 5? (x) is then binarized into (x(x) using a threshold 7, providing pixel-level supervision.
Initially, the dynamic parameter d of each Gaussian point is set to O, resulting in a dynamic value of (x(x) = 0.5 for each pixel. When the cross-entropy loss La is employed as the loss function, the Gaussian points that intersect with the dynamic pixel D2(x: will receive positive gradient, leading to the dynamic parameter d expanding towards + Conversely, when Gaussian points intersect with static pixels, the dynamic parameter d will expand towards -00. Due to the properties of the Sigmoid function, the dynamic parameter can extend infinitely towards both - oo and + allowing us to better distinguish between dynamic and static points.
When a Gaussian point intersects both dynamic and static pixels (e.g., in the presence of occlusion), it will receive two opposing gradient values. If the positive gradient is larger, its dynamic value will be greater than O, classifying as dynamic point. Conversely, if the negative gradient dominates, it will be classified as static point.
14

---

Published as a conference paper at ICLR 2025
Table Per-scenes results on the NV3D dataset. The best and the second best results denoted by red and blue.
Method
Coffee Martini
Spinach
Cut Beef
Flame Salmon
Flame Steak
Steak
Mean
MixVoxele
29.36
31.61
31.30
29.92
31.21
31.43
30.80
NeRFPlayer
31.53
30.56
29.35
31.65
31.93
29.13
30.69
HexPlane
32.04
32.55
29.47
32.08
32.39
31.70
K-Planes
29.99
32.60
31.82
30.44
32.38
32.52
31.63
4DGS
27.34
32.46
32.90
29.20
32.51
32.49
31.15
3DGStream
27.75
33.31
33.21
34.30
33.01
31.67
SpaceTimeGS
28.61
33.18
33.52
29.48
33.64
33.89
32.05
Real-Time4DGS
28.33
32.93
33.85
29.38
34.03
33.51
Swirt)DLLiteOurs)
28.84
32.57
32.82
29.92
33.13
33.48
31.79
Swift4D(Ours)
29.13
33.05
33.80
29.75
33.67
33.98
32.23
Finally, we provide the formula for calculating the gradient received by each Gaussian point. Based on this formula, the CUDA code be easily written. Assuming need compute the dynamic value gradient of Gaussian point 9. the equation as following: ad, known grad1:
ôLs Due to autograd, we have
0Ld
grad
(10)
0 0(E,=1 d,a; -I
we only need to compute:
0 1 da IT==1 (1 aBr)
g-1
grad2
(a' II
(1 - (1-d)))
(11)
ddg
j=1
So, the final formula is as follows:
OLd 0dg
OLd
0 daa [1
(12)
aEEn da
- og)))
ddg
9-1 (ag
(1- af,)
(13)
a(Σ, da [1 -1 af))
From this formula, can be seen that the gradient the dynamic value is related to occlusion, self-opacity, and the distance to the camera plane, which is very reasonable.
A.3
MORE RESULTS
Fig II shows the rendering results from new viewpoints at different iteration of training. It can be observed that our method achieves very high quality after 7000 epochs (approximately 10 minutes), demonstrating that our approach is highly efficient for reconstructing 4D dynamic scenes. Fig [2 demonstrates that our method can effectively segment dynamic points. Tab. 5 presents the results of several static and dynamic methods on the basketball court datase
showing that our method outperforms 4DGS et al. (2024). To demonstrate the robustness and generalization of approach, we also conducted experiments the ENeRF dataset. The results, shown in Table 6 follow the training policies described in 4k4d Xu et al. 2024)..
15

---

Published as a conference paper at ICLR 2025
(a) GT
(b) 3000
(c) 7000 its
(d) 13000 its
Figure 11: Training Epoch Comparison: the results of method in 3000, 7000, 13000 epochs. Based on the results from 3000 iterations, our method demonstrates rapid convergence.
Table 5: Quantitative comparison on the Basketball court dataset. The first four methods correspond to static methods, tested on the first frame, while the last two methods represent dynamic methods, tested on 20 frames.
Method
PSNR t
SSIM 1 4
LPIPS
Gof Yu et al. (0024b
30.39
0.949
0.141
2DGS Huang et al. (2024a
30.78
0.949
0.187
PixelGS ZZang et al.
(2024
29.26
0.946
0.168
3DGs et al.
2023)
30.50
0.949
0.171
WDGSWW et al. (2024)
27.87
0.921
0.191
Swilt
29.03
0.933
0.187
16

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
(d)
Figure 12: Decomposition results. (a) is the dynamic-static pixel mask, (b) the dynamic map rendered with our dynamic value d of Gaussians, (c) is the image rendered with dynamic Gaussians, and (d) the GT image.
Table 6: Performance comparison of different methods ENeRF dataset. The results are derived from 4k4d.
Methods
PSNR
SSIM
LPIPS 1
ENeRF Lin 2022
25,452
0.809
0.273
IBRNet Wang et al. (2021)
24.966
0.929
0.172
KPlanes Fridovich- Keil et al. (2023)
21.310
0.735
0.454
4k4d Xu et al. 2024)
25.815
0.898
0.147
Swift4D (Ours)
26.12
0.911
0.070
17

---

Mean: du
ReLU
N x 3
333
3
Opacity: do
ReLU
Nx1 1
Color: dsh
N X 128
ReLU
N X 48
x
t
4D Hash
48
48
Rotation: dr
N x 4
ReLU
4
4
Scaling: ds
N 3
ReLU
333
3
Figure 13: MLP Structures. For each dynamic point, we use five small MLPs predict the deformations.
Figure 14: The result of hierarchical rendering in dynamic scenes based on the dynamic parameter d of Gaussian points. The specific video can be found in the supplementary material.
Figure 15: The training results of the basketball court from four novel viewpoints. The images above are the GT images, and the ones below are our rendered results.
18


Reviews:
[{"review_id": "54C1qQEoGO", "reviewer": "Reviewer_9PWz", "summary": "The sources describe Swift4D, a novel method for reconstructing dynamic scenes by separating static and dynamic elements and using a 4D hash grids to model the temporal information of dynamic Gaussian points. The separation improves efficiency, reducing both storage requirements and training time, while achieving strong rendering quality in various dynamic scene scenarios. The authors test their method on  N3DV, MeetRoom and Basketball court datasets and compare to different recent works.", "questions": "- Please comment on the  Weaknesses above.\n- How can the model differentiate between specular reflections and dynamic elements?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The paper is overall well written and easy to understand. The authors provide detailed descriptions of the methodology and experiments.\n- According to the quantitative evaluation, the method performs favourably to competing approaches in image quality as well as optimization time.\n- The idea is relatively simple of static and dynamic decomposition, but appears to be effective in terms of speed.", "weaknesses": "- A key component of this work is the dynamic and static decomposition. The method introduces a dynamic level parameter that is optimized for each gaussian. As explained in section 3.2, a 2D dynamic-static pixel Mask is computed from the training video by leveraging the temporal standard deviation of each pixel. However, since the video can have camera motion as well as dynamic elements that yield varying the pixel intensity it is unclear to me how the standard deviation can serve as good criterion to  differentiate dynamic scene elements from static ones. Does this criterion work well on videos with strong camera motion?\n- The authors propose a temporal pruning strategy, it would be good to discuss at some point existing pruning strategies, e.g. LightGaussians (Fan et al. NeurIPS 2024), and the relation ship the proposed pruning method.\n- In the supplementary material videos, there seem to be artifacts of vanishing dynamic elements. In 2_baskball2.mp4 some players vanishing from frame to frame and it appears to have temporal inconsistencies, that questions the effectiveness of the method. Is this a common issue?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q0Qdi3b4hw", "reviewer": "Reviewer_TAqw", "summary": "This paper proposes Swift4D, which extends 3DGStream with a static & dynamic decoupling strategy. Doing this significantly reduced the model size and improved the rendering quality. The authors also introduce other tricks like  spatio-temporal structure modeling and adjusted density control.", "questions": "As described in weakness, my questions mainly focus on the generalizability to texture-less regions, highly dynamic videos (with different static-dynamic compositions), and long-term videos.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The motivation and technique design for this paper is straightforward and reasonable. Decoupling static and dynamic contents is an intuitive approach when dealing with real dynamic scenes. Although the approach is questionable, it may help researchers to understand the problem better and leverage the explicit structure of 3DGS.\n* The experiment results on different datasets show improvements in various aspects, like training time and PSNR, though the FPS is lower, and this is not an on-the-fly method.\n* The writing and illustration of this paper is pretty clear to me.", "weaknesses": "* One critical weakness is the implementation of decoupling static and dynamic parts. Specifically, they use 2D supervision to build the dynamic factor. However, the way they build 2D masks is just to calculate the temporal variance of each pixel, which should lead to problems when dynamic movements happen in textureless regions. These regions can easily be recognized as static parts. Such a phenomenon could be observed in the sear_steak_static result of the supplementary, where part of the cloth is classified as static. In addition, calculating the variance throughout the entire time could cause problems in long-sequence video.\n* It seems that this work can't deal with single-view videos like DNeRF. This could be a general limitation as it's hard to deploy and compare with other methods that can deal with such cases (e.g., SCGS). In addition, it needs to be trained offline, making the comparison with 3DGSStream kind of unfair.\n* To me, the result in Fig.7 does actually demonstrate the sensitivity of the dynamic threshold, where it affects the PSNR a lot. I am worried about the generalizability of the method since it's tested on scenes with similar dynamic-static compositions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GBaYbGHWGE", "reviewer": "Reviewer_c4Zw", "summary": "The paper introduces a 4D Gaussian representation for dynamic scenes, utilizing a learned deformation field to model Gaussian dynamics. To reduce the redundancy in dynamic learning, the method employs a decomposition strategy to separate dynamic and static elements. For the dynamic components, 4D spatiotemporal coordinates query a 4D hash grid, retrieving features from multi-resolution grids. An MLP then predicts the residual values of Gaussian attributes to capture the dynamics. By isolating dynamic from static components, this streamlined model efficiently handles dynamic modeling while occupying less space.", "questions": "I hope the author can respond to my concerns and suggestions mentioned in the weaknesses section. If there are any issues with the experimental results, recommended citations, or comparisons, I would appreciate an explanation.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Decomposing static and dynamic elements is beneficial, effectively reducing the model's learning burden and increasing inference speed, as shown in Table 3. Notably, this separation is also present in the concurrent work GauFRe (Liang et al.) and the published work SC-GS (Huang et al.). This concept is a major contribution to GauFRe and a detailed implementation in SC-GS. Citing these methods when discussing this idea is appropriate.\n\n2. The pruning strategy in Section 3.4 is innovative. Gaussian Splatting and NeRF, based on Multi-View Stereo (MVS) principles, converge when multi-view images reach a photometric consensus. An isolated Gaussian point, recognized by only a few views, tends to become a floating artifact, causing overfitting. This insightful point is valuable and inspires the community.\n\nThe two points already make me inclined to accept the paper, but there are some concerns. Please refer to the weaknesses section.", "weaknesses": "The experimental results and citations are insufficient. The FPS of Swift4DLite (Ours) is missing in Table 1, with no explanation provided. Additionally, the rendering speed is relatively slow, similar to 4DGS (Yang et al.) and SpaceTimeGS, with comparable rendering quality. The training time for SpaceTimeGS reported in Table 1 is too long; it should be under an hour. The FPS of SpaceTimeGS is also lower than I expected (130+). Moreover, several closely related works are not compared or cited:\n\n1. Huang, Yi-Hua, et al. \"SC-GS: Sparse-controlled Gaussian splatting for editable dynamic scenes.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n2. Yang, Ziyi, et al. \"Deformable 3D Gaussians for high-fidelity monocular dynamic scene reconstruction.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n3. Lin, Youtian, et al. \"Gaussian-flow: 4D reconstruction with dynamic 3D Gaussian particles.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nI recommend at least including comparisons with (1) and (3), as these methods model Gaussian dynamics using spatially and frequency-compact bases, respectively. Additionally, SC-GS achieves high FPS in rendering images by inferring the motion of sparse control points, according to their Supp.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wrDJ7tURqr", "reviewer": "Reviewer_BxBK", "summary": "This paper presents a Gaussian-splatting-based approach for dynamic scene reconstruction. It first builds a canonical space and decomposes the scene into static and dynamic components. Swift4D leverages a compact hash grid to model the deformations within the dynamic components. The proposed method includes a mechanism for identifying dynamic Gaussians and controlling density. Experimental results demonstrate that this method outperforms others.", "questions": "This paper has some merits. However, the scope and limitations of the proposed method are not clearly defined.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The authors present an innovative approach to decomposing scenes into static and dynamic components by introducing a novel Gaussian attribute combined with alpha blending. To the best of my knowledge, this is the first method to leverage Gaussian splatting for scene decomposition. The results demonstrate its effectiveness on the datasets used, achieving high-quality rendering.", "weaknesses": "The primary concern regarding this paper is the lack of sufficient comparisons. While the authors assert that their method is suitable for dynamic scenes, the datasets utilized predominantly feature forward-facing scenes with limited movements. Although the VRU dataset is a multi-view dataset that captures inward-looking scenes, the authors did not provide quantitative comparisons involving it. Additionally, there are other multi-view datasets [1, 2] on which Swift4D could be evaluated, particularly D-NeRF [1], which has been employed in various deformation-based dynamic neural rendering methods that could serve as points of comparison.\n\nThe authors mention other methods, Liang et al. (2023) and He et al. (2024), that are also doing static-dynamic decomposition. However, it would be more beneficial to compare their methodologies with the proposed method through quantitative experiments rather than relying solely on qualitative descriptions.\n\nFurthermore, the video results from the VRU dataset exhibit noticeable abrupt scene changes, likely due to the segmentation processing of the video. The authors should address this phenomenon in the paper.\n\n\n[1] Pumarola, A., Corona, E., Pons-Moll, G., & Moreno-Noguer, F. (2021). D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 10318-10327).\n[2] Xu, Z., Peng, S., Lin, H., He, G., Sun, J., Shen, Y., ... & Zhou, X. (2024). 4k4d: Real-time 4d view synthesis at 4k resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 20029-20040).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)