DN-4DGS: Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering
Authors: Jiahao Lu
Jiacheng Deng
Ruijie Zhu
Yanzhe Liang
Wenfei Yang
Xu Zhou
Tianzhu Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic Scene Rendering
3D Gaussian Splatting
Point Cloud
Primary area: machine_vision

Abstract: Dynamic scenes rendering is an intriguing yet challenging problem. Although current methods based on NeRF have achieved satisfactory performance, they still can not reach real-time levels. Recently, 3D Gaussian Splatting (3DGS) has garnered researchers' attention due to their outstanding rendering quality and real-time speed. Therefore, a new paradigm has been proposed: defining a canonical 3D gaussians and deforming it to individual frames in deformable fields. However, since the coordinates of canonical 3D gaussians are filled with noise, which can transfer noise into the deformable fields, and there is currently no method that adequately considers the aggregation of 4D information. Therefore, we propose Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering (DN-4DGS). Specifically, a Noise Suppression Strategy is introduced to change the distribution of the coordinates of the canonical 3D gaussians and suppress noise. Additionally, a Decoupled Temporal-Spatial Aggregation Module is designed to aggregate information from adjacent points and frames. Extensive experiments on various real-world datasets demonstrate that our method achieves state-of-the-art rendering quality under a real-time level. Code is available at https://github.com/peoplelu/DN-4DGS.

DN-4DGS: Denoised Deformable Network with Temporal-Spatial A Aggregation for Dynamic Scene Rendering
Jiahao Lu
Jiacheng Deng'
Ruijie Zhu²
Yanzhe Liang' Xu Zhou
Wenfei Yang!.² Tianzhu Zhang
University of Science and Technology of China/Deep Space Exploration Lab, "Jaanghuai Advance Technology Center, Sangfor Technologies Inc. (luj iahao, dengje, ruijiezhu, yzliang) @mail ustc. edu. cn, fyanguf, tzzhang) @ustre edu zhouxu@sangfor. com. en
4DGaussian
Ours
4DGaussian
Ours
27.29
28.50
25. 10
(a) PlenoptieVideo
(b) HyperNeRF
Figure 1: (a) The visualization results on Plenoptic [ dataset. (b) The visualization results on HyperNeRF [ dataset. The numbers below the images represent PSNR.
Abstract
Dynamic scenes rendering is an intriguing yet challenging problem. Although current methods based on NeRF have achieved satisfactory performance, they still not reach real-time levels. Recently, 3D Gaussian Splatting (3DGS) has garnered researchers' attention due to their outstanding rendering quality and real-time speed. Therefore, new paradigm has been proposed: defining a canonical 3D gaussians and deforming to individual frames deformable fields. However, since the coordinates of canonical 3D gaussians filled with noise, which can transfer noise into the deformable fields, and there currently no method that adequately considers the aggregation of 4D information. Therefore, we propose Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering (DN-4DGS). Specifically, Noise Suppression Strategy is introduced to change the distribution the coordinates of the canonical 3D gaussians and suppress noise. Additionally, a Decoupled Temporal-Spp -apttral-Spatial Aggregation Module is designed to aggregate information from adjacent points and frames. Extensive experiments
various real-world datasets demonstrate that our the art rendering quality under real-time level. Code is available https:/giithh.ccmmmppllluuDDD--D
method achieves of -aeeee--ffhh-eaaae
"Corresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Canonical
Deformable 3D Guusslans (4DGonssian)
Canonical
Deformable
Deformable
Gausslans
Gaussians
Gaussiuns
3D Gunsslans
(4DGaussian)
(Ours)
Stage (Ours)
Stage 2 (Ours)
Figure 2: Comparison of our render visualization with 4DGuussian [11]. The results are rendered
on HyperNeRF 2 dataset and use the point cloud provided by HyperNeRF for Gaussian initialization (Sparse Init). Image I: canonical 3D gaussians generated by 4DGaussian. Image 2: deformable 3D gaussians generated by 4DGaussian. Image 3: canonical 3D gaussians generated by method. Image 4: deformable 3D gaussians after the first stage. Image 5: deformable 3D gaussians after the second stage. Image 6: ground truth. The yellow box emphasizes that through two-stage deformation process, our method can produce higher-quality rendering results.
1 Introduction
Dynamic scene reconstruction from single or multi-view videos is a crucial task in computer vision, with applications such as VR/AR 4 3D perception [5 77, movie production [8) etc. Neural Radiance Fields (NeRF) [9] offer promising approach by representing scenes with implicit functions derived from multi-view inputs. By incorporating time as an additional input [TO 11 NeRF enables dynamic scene rendering. However, the original NeRF model suffers from significant training and rendering costs, attributed to the high number of points sampled per camera ray and volume rendering.
Recently, the emerging 3D Gaussian Splatting (3DGS) [12 has significantly increased rendering speed real-time level compared NeRFss by employing differentiable rasterizer for 3D Gaussian primitives. 3DGS directly optimizes the parameters of 3D gaussians (position, opacity, anisotropic covariance, and spherical harmonics (SH) coefficients) and renders them through projection and a-blending. Given the explicit expression nature of 3DGS, recent studies 13 represent dynamic scenes by defining canonical 3D gaussians and deforming it to individual frames in deformable fields. Specifically, during the execution of the deformation field, the coordinates ryz along with time are used as input, and the output corresponds to the changes in Gaussian properties. It is noteworthy that the existing methods, when designing deformable networks, either directly map the 4D coordinates of each input point to the latent space using MLPs [13 14], or use HexPlane [15] to interpolate a series of learnable embeddings to obtain the latent of each point 11]
Both of these approaches have drawbacks that can be ignored. 1) Canonical 3D gaussians are synthesized from mulli--frame images of dynamic scenes. Due to the presence of dynamic regions and the specific design of A (canonical 3D gaussians) B (deformable network), canonical 3D gaussians exhibit significant noise, as illustrated in Figure 2 This noise is inevitably transferred to the deformable field after the input zyz passed through the deformable network. To elaborate on the "Noise": In the canonical + deformable design, we input the canonical Gaussian coordinates zyz and time into the deformable network. The deformable network essentially performs basic operations (addition, subtraction, multiplication, division) on the coordinates zyz and time (. Since the point-to- point relationships within the canonical Gaussians are chaotic and erroneous, as shown in Figure 2 it is predictable that feeding these erroneous coordinates into the deformable network will transfer this error into the deformation field, introducing inaccuracies in the final deformations Ar, Ay, Az. 2) There is lack of feature aggregation for
information, yet due to the presence of noise in canonical 3D gaussians' zyz, direct feature aggregation for spatial information would further amplify noise, affecting the learning of the deformable field. Therefore, spatial aggregation after denoising is very crucial.
To address the aforementioned issues, we propose Denoised Deformable Network with Temporal- Spatial Aggregation for Dynamic Scene Rendering (DN-4DGS), primarily consisting of two com- ponents: the Noise Suppression Strategy (NSS) and the Decoupled Temporal-Spatial Aggregation Module (DTS). To address the initial issue, the design NSS incorporates two deformation oper- ations. The first deformation operation is standard deformation. It takes the coordinates zyz of

---

the canonical 3D gaussians and time t as input and outputs corresponding coordinate deformations Az, Ay, Az. The second deformation builds upon the first by adding År, Ay, Az to the original syz, creating modified set of coordinates that is then input into new feature extraction network. This strategy achieves a successful
The entire process is illustrated by image 3, 4 and 5 in Figure 2 alteration of the distribution of coordinates zyz through the initial coordinate deformation, resulting in noise reduction and the generation of a more accurate deformation field. It's worth noting that during the early stage of training, we only perform the first deformation operation. Only after achieving acceptable results with the first deformation operation do we proceed to the second deformation operation to further enhance accuracy. To address the second problem, we design the DTS. The reason we decouple apatial-temporee aggregation is due to the presence of noise in the coordinates of the canonical 3D gaussians. If we directly perform spatial aggregation on the coordinates of the canonical 3D gaussians, the noise information is inevitably amplified after a series of aggregation operations such as k-nearest neighbors (KNN) [66.. significantly affecting the results of the defor- mation field. Therefore, based on the first design NSS, we conduct spatial aggregation during the second deformation operation. Considering that temporal information is unrelated to the canonical 3D gaussians, temporal aggregation can be directly incorporated into the first deformation operation enhance feature extraction capabilities. In order to reduce computational overhead and considering that temporal information has already been effectively extracted in the first deformation operation, we do not perform temporal aggregation in the second deformation operation. In conclusion, our main contributions are outlined as follows:
(i) We introduce novel representation called Denoised Deformable Network with Samporal--pattid Aggregation for high-fidelity and efficient dynamic seene rendering.
(ii) We promose the Noise Suppression Strategy, which can change the distribution of the coordinates of the canonical 3D gaussians, suppress noise and generate a more precise deformation field.
(iii) We promose the Decoupled Temporal-Spatial Aggregation Module to aggregate information from adjacent points and frames.
(iv) Extensive experiments on various real-world datasets demonstrate that our method achieves state-of-the-arr rendering quality under a real-time level.
2 Related Work
Dynamic NeRF Novel view synthesis has been hot topic in academia for several years. NeRF 9 models static scenes implicitly using MLPs, and numerous studies [17 19, 2 TO 20, 21] have ex- tended its application to dynamic scenes through canonical 3D grid structure and a deformation field. HyperNeRF (2) models object topology deformation using higher-dimensional inputs, while DyN-
eRF 1 employs time-conditioned NeRF to represent a 4D scene. However, these approaches, based on vanilla NeRF, suffer from high computational costs due to ray point sampling and volume render- ing. To address this issue, several acceleration methods (22 23
24 25
26
27
[5] 28 29 30 31 32]
have been proposed for rendering dynamic scenes. VR (2 introduces grid representation, while IBR-based methods (24 [5] utilize multi-camera information for improved quality and effi- ciency. TensorRF (33) adopts multiple planes as explicit representations for direct dynamic scene modeling. Recent approaches such as K-Planes 34 Tensor4D 30, and HexPlane [15] have also been proposed. NeRFPlayer (35) introduces a unified streaming representation for both grid- based and plane- methods, utilizing separate models to differentiate static and dynamic scene compo- nents, albeit at the cost of slow rendering times. HyperReel (36] suggests a flexible sampling network coupled with two planes for dynamic scene representation. Despite the improvements in training and rendering speed achieved by these methods, they still fall short of meeting real-time requirements.
Dynamic Gaussian Splatting, Recently, 3D Gaussian Splatting (3DGS) [12 has garnered increasing attention from researchers due to its superior rendering quality and real-time rendering speed. The method employs soft point representation with attributes including position, rotation, density, and radiance, and utilizes differentiable posnt-based rendering for scene optimization. Soon after, several concurrent works [I] B 14 37 have adapted 3D Gaussians for dynamic scenes. These methods represent dynamic scenes by establishing canonical 3DGS and deforming it to individual frames using deformable fields. Yang et al. [13] predict per-Gaussian offsets using an additional MIP on canonical 3D gaussians, while Wu et al. [I] substitute the MLP with multi-resolution HexPlanes [55] and lightweight MIP. Our work introduces the Noise Suppression Strategy to change the distribution of the coordinates of the canonical 3D gaussians and generate more precise
3

---

C. Conden
Oh Seeelre
Figure 3: The overall framework of our method DN-4DGS. Our approach employs a two-stage deformation process. In the first deformation, the well-designed Temporal Aggregation Module is utilized to aggregate temporal information. After the first deformation, the coordinate distribution of 3D gaussians is altered, and noise is suppressed. Subsequently, proceed with the second deformation, utilizing the Denoised Spatial Aggregation Module to aggregate spatial information.
deformation field. Additionally, for better aggregation of temporal-spanee information, propose the Decoupled Temporal-Spp Aggregation Module to consolidate information from adjacent points and frames.
3 Preliminary: 3D Gaussian Splatting
Given images at multiple known viewpoints and timesteps, 3D Gaussian Splatting (3DGS) [12 optimizes set of attributes (position, opacity, anisotropic covariance and spherical harmonics) via differentiable rasterization. 3DGS realize high-fidelity rendering of static 3D scenes in real-time.
Suppose a 3D Gaussian G(i) has the following attributes: position Σ, and spherical harmonics h;- The covariance matrix E, is decomposed Σ; RRSTTR for optimization, with R as a rotation matrix represented by quaternion & E SO(3). and S as a scaling matrix represented by 3D vector Each Gaussian has an opacity value to adjust its influence in rendering and associated with sphere harmonic (SH) coefficients h for view-dependent appearance. The final opacity of 3D gaussian any spatial point can be represented
μ4, opacity di covariance matrix
a = oge 1 (=--) (2-mi.)
To render a 2D image, 3D gaussians are projected to 2D space and aggregating them using fast a-blending. The 2D covariance matrix and center D&D JWEWT and n JW p4. The color C(u) of a pixel u is rendered using the fast a-blending operation:
C(u)= Σ SH,, SEN
22)
where 7, = 11; -1yy), SH is the spherical harmonic function and Us is the view direction.
Method
4.1 Overview
Our goal is to reconstruct dynamic 3D scenes from single/multi-view videos. Following previous works [3], we represent the geometry and appearance of the dynamic scene using canonical 3D gaussians and model the motion through the deformation fields. An overview of our method is shown in Figure 3 We first describe the details of the Noise Suppression Strategy (NSS) in Section 4.2
Then in Section 4.3 we present the design of the Decoupled Temporal-Spatin Aggregation Module
(DTS). Section 4.4 details our optimization process.
4.2 Noise Suppression Strategy
In this section, we attempt to mitigate the terrible noise of the canonical 3D gaussians, shown in Figure Specifically, the Noise Suppression Strategy comprises two deformation operations. The first deformation operation is a standard deformation. It takes the coordinates y, of the canonical 3D gaussians and time / as input and outputs corresponding coordinate deformations Ar, Ay, Az. To simplify, here we only list the attributes y. z, while other attributes are detailed in the subsequent Section
Az, Ay, Az (z Z.)).

---

where V/ represents the first deformation operation, Tn represents the set of t's neighbors. The details of V are introduced in Section 4.3.1 Next, after obtaining Ar, Ay. Az, we add them the original Jr y. 2.
x' ,' 2 = x+ Ar,y + Ay; = + Az.
Following this, the second deformation operation is carried out.
Ar', Ay', A=' V (Pk, 1),
(5)
where V represents the second deformation operation, Ps represents the set of k neighbors of (a', z²). The details of vp' are introduced in Section 4.3.2 In this deformation, due to the successful alteration of the input coordinate distribution during the first deformation stage, we can obtain more accurate Gaussian positions compared to the canonical Gaussian. As a result, the noise in the input is attenuated.
Overall, Noise Suppression Strategy (NSS) is strategy that uses two stages of deformation to reduce the impact of noise on the deformable network. During this process, we have two training phases. In the early training phase, we only supervise the first deformation. Once the Gaussian coordinates obtained from the first deformation are relatively accurate, we add the second deformation and shift the supervision to
4.3 Decoupled Temporal-Spatial Aggregation Module
Local feature aggregation is very important for 3D point clouds, which can effectively extract local structure information. PointNet++ [16] introduces the set abstraction layer to aggregate information from spatially adjacent points, which has become a fundamental operation in various point cloud tasks [38] 39 40 41 42). Therefore, to enhance the accuracy of the deformation fields, an intuitive approach is to perform neighbor aggregation for each gaussian.
For 4D gaussians, there are four dimensions of information æ, V, f available for aggregation. As discussed in the introduction, performing local aggregation on noisy coordinates would further am- plify the noise. Therefore, for 2, V. 2, spatial aggregation is conducted during the second deformation operation. Since temporal information is unrelated to the canonical 3D gaussians, temporal aggre- gation can be directly integrated into the first deformation operation enhance feature extraction capabilities. To reduce computational overhead, and considering that temporal information has already been effectively extracted in the first deformation operation, we omit temporal aggregation in the second deformation operation. The entire process is referred to as decoupled temporal-spatie aggregation.
4.3.1 Temporal Aggregation Moudle
For each gaussian G,()), we first input x. y, 2, into the Feature Encoding. Regarding Feature Encoding, we can utilize the MLPs from D3DGS [33 or the HexPlanes from 4DGaussian (TT) After that, we acquire F((1) at time L. Next is a critical step, where we repeat the above- mentioned process obtain the features Fi-1 (i) for time t- and the features F (i) for time t + 1. It's worth noting that here, the "1" represents one timestep. Next, similar to PointNet++ [16), we perform the aggregation operation. As illustrated in Figure 4 we merge ( Fi()), and (i) together to form F (i) E 3x31xX and input into a lightweight MLP for channel change. Then, we perform MaxPooling along the first dimension of F(i) to generate (i) E 121x Additionally, we introduce attribute )d. which is learnable embedding. This attribute can provide information independent of coordi- nates without interference due to adjacency. We have also observed similar design in recent work E-D3DGS 43). Finally, Fmux (i), Ei) and D, are concatenated together to generate deformation:
[Pmxx (i), Fę(1))
MaxPooling
MLP
Fi-1())
Fe(1)
Fi+1(1)
Figure 4: The structure of aggregation operation.
F.(1)( [Fmax (i), F.), Fg F.(1) + (Ar, Ay, Az, Ar. As, Ao. Ah),
(6)
(7)
5

---

Plenoption Video
HeperNeRF
Dense Enit
Sparse Init
Figure 5: More rendering images of canonical 3D gaussians. Here, Sparse Init refers to using the point cloud provided by the HyperNeRI [2] dataset (COLMAPsPM) SFM (44) for Gaussian initialization, while Dense Init denotes generating denser point cloud via COLMAP 44 In fact, Dense Init can produce better rendering quality, but due to the need for regenerating, it consumes more computational resources.
where Fø is the deformation MLP head, r is rotation quaternion, is vector for scaling, is an opacity, and SH coefficients for modeling view-dependent color.
4.3.2 Denoised Spatial Aggregation Moudle
After obtaining the new y', 22,, we input them into the denoised spatial aggregation module for spatial aggregation. Concretely, calculate the k-nearest neighbors for each gaussian based on v' z'. Then, we aggregate the features of the k-nearest neighbors to obtain (a( E R RRIXXxCs
MaxPooling is then performed on (i) to get F. Finally, F, (x), Fim (i) and x' '',,/ v/ concatenated together to generate the second deformation:
Fi(()' (F,( Fm x. ='},
(8)
Fo (X) + (Az, Ay. Az, Ar, As, Ao, Ah),
(9)
where Fg is the deformation MLP head in the second deformation operation.
Overall, Decoupled Temporal-Spatial Aggregation Module (DTS) is specific feature aggregation method we propose. Unlike 4DGaussian, D3DGS lacks explicit spatiotemporal aggregation, so we designed DTS to aggregate spatiotemporal information. Considering that inaccurate coordinate rela- tionships would be amplified through spatial aggregation (KNN), we only perform spatial aggregation in the second stage of NSS, which is DSAM. We name the spatial aggregation module "Denoised Spatial Aggregation Module" (DSAM) because the Gaussian coordinates input into DSAM are more accurate (denoised), shown in the fourth column of Figure 9 Therefore, we prefix the Spatial Aggregation Module with "Denoised". DSAM itself does not have denoising capabilities; it solely performs spatial feature aggregation.
4.4 Optimization
The parameters be optimized include the deformable network and the attributes of each 3D gaussian G(i): Pi, Fs, D. f4 and Ve Following 4DGaussian [II]. we use the reconstruction loss L1 and
gird-based TV loss [15 45 34 46) Lrv supervise the training process. Additionally, add
D-SSIM term Lesim to improve structural similarity:
L= XC1 + (1 (1-A)) Ltvs
(10)
where is the hyperparameter, It is worth noting that we employ a two-stage training strategy. During the early stages of training, we exclusively execute the first deformation operation. Once satisfactory results are attained with the initial deformation operation, we then proceed to implement the second deformation operation further refine accuracy. The reason for this strategy is that only the deformation Az, Ay. Az in the first stage is sufficiently precise to remove large amount of noise, thereby positively impacting the deformation in the second stage.
Experiment
5.1 Experimental Setup
Dataset and Metrics. Plenoptic (0] dataset includes 20 multi-view videos, with each scene consisting of either 300 frames, except for the flame salmon scene, which comprises 1200 frames.

---

Method
PPSNR(1) SSIM(T) LPIPS(.)
Time(L)
[FPS(1) Storago(MB)C) (B)
DyNeRF 1
29.58
0.099
1344 hours
0.01
28
NeRPPlayer
35
30.69
0.909
0.111
6 hours
0.045
HyperReel 36
31.10
0,921
0.096
9 hours
2.0
360
HexPlane-aie all* 15
31.70
0.984
0.075
12 hours
0.2
250
KPlanes
33
31.63
0.964
1.8 hours
0.3
309
4DGS
31.19
0.940
0.051
9.5 hours
19.5
8700
E-D3DGS 43
31.31
0.945
0.037
2 hours
43.1
35
4DGaussian []]
31.15
0,940
0.049
40 mins
30
90
Ours
32.02
0.944
0.043
50 mins
15
112
Table 1: Quantitative comparison on PlenopticVideo dataset. We display the average
PSNR/SSIM/LPIPS (Alex) metrics for novel view synthesis on dynamic scenes, with each cell colored to indicate the best second best and third best
These scenes encompass a relatively long duration and various movements, with some featuring multiple objects in motion. We utilized Plenoptic Video dataset to observe the capability to capture dynamic areas. Total six scenes (coffee martini, cook spinach, cut roasted beef, flame salmon, flame steak, sear steak) are utilized to train and render. Rendering resolution set to 1352 x 1014 handheld capture rig. We train and render on four scenes (3D Printer, Banana, Broom, Chicken) at a resolution downsampled by factor of two to 540 x 960. NeRF-DS 47 dataset consists of seven captured videos (Sieve, Press, Plate, Cup, As, Bell, Basin) with camera pose estimated using colmap [44]. The dataset involves variety of rigid and non-rigid deformation of various objects. We train and render on the seven scenes. Rendering resolution is set to 480 x 270.
Video
HyperNeRF [2] dataset includes videos using two Pixel 3 phones rigidly mounted
We report the quality of rendered images using PSNR, SSIM 48 MS-SSIM and LPIPS [4 Higher PSNR, SSIM and MS-SSIM values and lower LPIPS values indicate better visual quality. To Plenoptic Video dataset, we report PSNR, SSIM and LPIPS (Alex). To HyperNeRE dataset, we report PSNR, SSIM and MS-SSIM. NeRF-DS dataset, we report PSNR, MS-SSIM and LPIPS (VGG).
Implementation Details. We train our model on a single RTX3090. The optimizer we utilize is Adam [50]. The learning rate is initially set at 1.6e-4, gradually decreasing exponentially to 1.6e-6 by the end of the training process. The learning rate for the voxel grid is initialized at 1.6e-3 and exponentially decays 1.6e-5, For hyperparameters, we tune K, as 16, 0.9 respectively, More details will be shown in the Appendix.
5.2 Comparison with existing methods.
Results on PlenoptieVideo. Table reports the results on Plenoptic dataset. Refer to the Appendix for per-scene details. Due to the incorporation of the Noise Suppression Strategy, which alters the distribution of canonical 3D gaussians coordinates to suppress noise, as well as the utilization of the Decoupled Temporal-Spatial Aggregation Module for feature aggregation, our approach demonstrates superior reconstruction quality across all metries compared to the baseline (4DGaussian [II). In fact, PSNR and LPIPS are currently the state-of tfate-of-the-ar art metrics. As the table shows, despite our method involving two stages of deformation, resulting in slight weakening in training time and FPS, it overall meets the requirements for rapid training and real-time demands. Regarding storage, due to the presence a new attribute ),, it may slightly exceed the baseline. To vividly illustrate the differences between our method and others, we visualize the qualitative results in Figure 6 From the regions highlighted in red boxes, is evident that our method can render higher-quality images.
Results on HyperNeRF. T reports the results on HyperNeRE dataset. In this dataset, we present results for both Sparse Init and Dense Init. Sparse Init refers to using the point cloud provided by the HyperNeRF dataset (COLMAPsPM) SFM (44) for Gaussian initialization, while Dense Init denotes generating denser point cloud via COLMAP MVS 44). From Table it can be observed that our method outperforms other Gaussian- based methods under both Sporse Triit and Dense Inii settings. Moreover, under the Dense Inil setting, our method achieves the current state-of-the-a performance. More importantly, we find that 4DGaussian is highly sensitive to the sparsity or density of Gaussian initialization. In contrast, approach benefits from noise suppression and feature
7

---

4DGaussian
Ours
GT
Coffee Martini
Flame Salmon
Figure 6: Qualitative comparisons on Plenoptic Video Dataset.
Method
PSNR(t)
SSIM(1)
MS-SSIM(1)
Time(L) [FP((1) Storage(MBB)()
Nerfies
19
22.23
0.803
hours
<1
HyperNeRF DS
2
22.2
0.598
0.811
32 hours
<1
-iNeuVVo--B
45
24.30
0.616
0.837
30 mins
48
D3DGS}
13
21.50
hours
10
18
4DGaussian
||
21.80
0.573
0.710
50 mins
38
11
Ourst
23.31 (+1.51) 0.618 (+0.045)
0.768 (+0.058)
1.1 hours
24
12
D3DGS*
23.43
3.5 hours
88
4DGaussian*
25.20
0.682
0.845
I hour
34
61
Ours
25.59 (+0.39) 0.691 (#0.009) 0.863 (+0.018) 1.2 hours
20
68
Table 2: Quantitative comparison on HyperNeRF dataset. Here, $ represents that we train the model based on Sparse Tnít.. & represents that we train the model based on Dense Init.
aggregation, resulting in a more pronounced performance improvement under the sparse setting. The qualitative results be observed from Figure From the regions highlighted in red boxes, method can render higher-quality images, as further supported by PSNR in gray cells.
Results NeRF-DS. Table presents the results NeRF-DS dataset. Our proposed method achieves better performance compared to previous methods, demonstrating the effectiveness and generalization of our method. More qualitative results shown in the Appendix.
Results on D-NeRF, As illustrated in the Figure 8 we have visualized the canonical results for both 4DGaussian and D3DGS. The results show that D3DGS has less noise compared to 4DGaussian, noise still present in moving areas. The quantitative results on D-NeRF dataset, as shown in the Table 3 indicate that our method improves performance both 4DGaussian and D3DGS, with enhancements on 4DGaussian surpassing those on D3DGS. Regarding the parameters of the deformation networks, in 4DGaussian: Most of the parameters are composed of the HexPlane and deformation head. Our method introduces an additional deformation operation, resulting in a slight increase the number of parameters compared to the baseline, with increase of only 0.03M parameters. In D3DGS: The network primarily consists of an MLP-based deformation network. To reduce the computational load, method halves the number of MLP layers, resulting in fewer overall parameters compared to the original D3DGS.
5.3 Ablation Studies
Evaluation of the model with different designs. To evaluate the effectiveness of proposed com- ponents, we conduct an ablation study in Table on Plenoptic Video dataset. Here, NSS, TAM, DSAM represents the Noise Suppression Strategy, Temporal Aggregation Moudle and Denoised

---

4DGaussian
Ours (Sparse Init)
4Gaussian (Desse Init)
Ours (Dense Init)
GT
(Sparse Init)
26.43
28.40
28.92
29.54 54
Chicken
24.12
26.69
19.16
29,90
Banana
Figure Qualitative comparisons on HyperNeRE Dataset. gray cells, the numbers represent PSNR.
Method
PSNR/()
SSIMI"")
Parameter
4DGusssian
4DGausian-Ours
34.53(-0.48)
0.981 (C-0.0024)
3/4(+-003)
D3DGS [3
39.51
0.5902
0,52
39.87(-0.36)
0.9922(40020)
Table Quantitative comparison on D-NeRF
Figure 8: The canonical results for both
dataset. Here, parameter refers to the parameters of the deformation networks corresponding different baselines.
4DGaussian and D3DGS.
Spatial Aggregation Moudle respectively. Specifically, the second row shows that with the use of two-stage deformation operations, our model can acquire a certain degree of improvement in quality. This highlights the meaning of noise suppression. The third row demonstrates that with the help of temporal aggregation, performance gain of 0.41, 0.02, 0.003 has been achieved in PSNR, SSIM and LPIPS. The fourth row demonstrates the effective collaboration between NSS and TAM, resulting in performance improvement. The fifth row indicates that if we do not utilize two-stage training, but instead only perform spatial aggregation on canonical 3D gaussians, an improvement in quality but also leads to a decrease. The sixth row indicates that if we replace TAM with an ordinary deformation network, there is slight drop in performance compared to last row. The last row indicates that if we combine all components together, the performance can reach its optimal level.
only fails bring about
Effectiveness of the two-stage deformation operations. To validate the significance of two-stage deformation operations, we conducted visual experiments. As shown in Figure D canonical 3D aussians exhibit a significant amount of noise, which severely affects the accuracy of the deformation field. To alleviate this issue, we employed two-stage deformation. As depicted in the fourth column of the figure, after the first deformation, there is a significant change in the distribution of coordinates ayz, effectively suppressing the noise. Moreover, due to design of temporal aggregation, corresponding PSNR value is even higher than that of 4DGaussian. Finally, by performing the second deformation operation on the basis of the first-stage deformation, the performance is further improved. This is attributed to the cleaner coordinates ryz and the design of spatial aggregation.

---

Method
PSNR(1)
SSIM(1)
LPIPS(()
NSS
TAM
DSAM
PSNR(())
SSIM(1)
LPIPS(L)
TiNeuVox 45
21.61
0.823
0.277
K
31.15
0,940
0.049
HyperNeRF
2
23.45
0.849
0.199
X
31.31
0.941
0.047
NeRF-DS 47
23.60
0.849
0,182
X
X 31.56
0.942
0.046
V
X
31.69
0.943
0.045
3D-GS 12
20.29
0.782
0,292
31.31
0.936
0.052
D3DGS 1
23.92
0.847
0.184
31.72
0.943
0.045
Ours
24.36
0.865
0.171
32.02
0.944
0.043
Table 4: Quantitative comparison on NeRF-DS
Table 5: Evaluation of the model with different
dataset.
designs on Plenoptic Video dataset.
17.71
24.10
16.62
25.79
26.84
Banana
13.28
25.05
13.01
25. 25.28
27 0
Chicken
Canonical
Deformable 3D Gaussians
Canonical
Deformable 3D Gaussians
Deformable 3D Gaussians
3D Gaussians
3D Gaussians
GT
(4DGousdan))
(4IKiousslan)
(Ours)
Stage
Stage (Ours)
Figure 9: Effectiveness of the two-stage deformation operations. In the gray cells, the numbers
represent PSNR.
5.4 Limitations and Future Work
Although two-stage deformation can alter the coordinate distribution of canonical 3D gaussians and reduce the noise introduced into the deformation field, the lack of simultaneous supervision for both stages 522 53 poses a challenge. Consequently, during the second stage, due to the lack of supervision in the first-stage deformation, the direction of coordinate deformation becomes uncontrollable to some extent. This, in turn, affects the spatial feature aggregation in the second stage. To address this issue, future work should explore the direction of simultaneous supervision.
Conclusion
In this paper, we introduce novel representation called Denoised Deformable Network with Temporal-Spatial Aggregation for Dynamic Scene Rendering. We promose the Noise Suppres- sion Strategy, which can change the distribution of the coordinates of the canonical gaussians, suppress noise and generate a more precise deformation field. To aggregate information from adjacent points and frames, we promose the Decoupled Temporal-Spatia Aggregation Module. Extensive experiments on various real-world datasets demonstrate that our method achieves state-of-the-art rendering quality under a real-time level.
Acknowledgements
This work was partially supported by the National Nature Science Foundation of China (12150007, 62306294), Dreams Foundation of Jianghuai Advance Technology Center (No. No.22023-2M0120119) and the Youth Innovation Promotion Association.
10

---

References
[1] Tianye Li, Mira Slavcheva, Michael Zollhoefer. Simon Green, Christoph Lassner, Changil Kim, Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patern Recognition, pages 5521-5531, 2022.
[2] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin- Bruall,, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologi- cally varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
[3] Hansung Kim, Luca Remaggi, Philip JB Jackson, and Adrian Hilton. Immersive spatial audio reproduction for vr/ar using room acoustic modelling from 360 images. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 120 126. IEEE, 2019.
[4] Rafael Pagés, Konstantinos Amplianitis, David Monaghan, Jan Ondiej. and Aljosa Smolić. Affordable content creation for free-viewpoint video and vr'ar applications. Journal Visual Communication and Image Representation, 53:192-201. 201, 2018.
[5] Jiahao Lu, Jiacheng Deng, and Tianzhu Zhang. Bsnet: Box-supervised simulation-assisted mean teacher 3d instance segmentation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20374-20384, 2024.
[6] Zhuoyuan Li, Yubo Ai, Jiahao Lu, ChuXin Wang. Jiacheng Deng, Hanzhi Chang, Yanzhe Liang, Wenfei Yang, Shifeng Zhang, and Tianzhu Zhang. Mamba24/8d: Enhancing global interaction in point clouds via state space model. arXiv preprint arXiv:2406. 17442, 2024.
[7] Jiacheng Deng, Jiahao Lu, and Tianzhu Zhang. Diffödetr: Agent- based diffusion model for semi cemi-supervised 3d object detection. arXiv preprint arXiv:2408. 00286, 2024.
[8] Kensuke Hisatomi, Kimihiro Tomiyama, Miwa Katayama, Yuichi Iwadate, Koji Matsunaga, Yoshiyuki Ito, and Wataru Ishihara. A method of vidco production using dynamic 3d models and its application to making scenes of crowd. SMPTE motion imaging journal, 118(7):99-36, 2009.
[9] Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99- (00,),)9-006,1200 2021.
[10] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 10318- 11118- 2021.
[11] Guanjun Wu, Taoran Yi. Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.088 08528, 2023.
[12] Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler, and George Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics (ToG), 42(4):1-14, 2023.
[13] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13100 2023.
[14] Yi-Hua Huang, Yang- controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.149333 2023.
Lang-Tian&o Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse-
[15] Ang Cao and Justin Johnson. Hexplane: fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages
2023.
[16] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. Pointnet++:: Deep hierarchical feature learning on point sets in a metric space. Advances in neural information processing systems, 30, 2017.
[17] Xiang Guo, Jiadai Sun. Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yumeng Zhang, and Jingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16022-16033, 2023.
[18] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.
11

---

[19] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman. Steven M Seitz, and Ricardo -ruall- Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865-5874, 2021.
[20] Edgar Tretschk, Ayush Tewari, Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 12959-12970 2021.
[21] Wenqi Xian, Jia-Bin Huang, Johannes Kopf, and Changil Kim. Space-timo neural irradiance fields for free-viewpoint video. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[22] Jia-Wei Liu, Yan-Pei Cao. Weijia Mao, Wenqiao Zhang. David Junhao Zhang, Jussi Keppo, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. Advances in Neural Information Processing Systems, 35:36762-366 36775, 2022.
[23] Zhengqi Li. Qianqian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. Dynibar: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[24] Haotong Lin, Sida Peng, Zhen Xu, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowci Zhou. Efficient neural radiance fields for interactive free-viewpoint video. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[25] Haotong Lin, Sida Peng. Zhen Xu, Tao Xie, Xingyi He, Hujun Bao, and Xiaowei Zhou. Im4d: High-fidelity and real-time novel view synthesis for dynamic scenes. arXiv preprint arX0v:23100088555, 2023.
[26] Stephea Lombardi, Tomas Simon, Gabriel Schwartz, Michael Zollhoefer, Yaser Sheikh, and Jason Saragih. Mixture of volumetric primitives for efficient neural rendering, ACM Transactions on Gruphics (ToG), 40(4): 2021.
[27] Sida Peng, Yunzhi Yan, Qing Shuai, Hujun Bao, and Xiaowei Zhou, Representing volumetric videos as dynamic mlp maps. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 4252 4222, 2023.
[28] Jiemin Fang, Taoran Yi, Xinggang Wang. Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[29] Sara Keil,, Giacomo Mcanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2479-12488. 2023.
[30] Ruizhi Shao, Zerong Zheng, Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16632-16643 2023.
[31] Feng Wang. Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song, and Huaping Liu. Mixed neural voxels for fast multi-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17706199166 2023.
[32] Liao Wang, Qiang Hu, Qihan He, Ziyu Wang, Jingyi Yu, Tinne Tuytelaars, Lan Xu, and Minye Wu. Neural residual radiance fields for streamably free-viewpoint videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 76-87, 2023.
[33] Anpei Chen, Zexiang Xu, Andreas Geiger, Jingyi Yu, and Hao Su. Tensorf: Tensorial radiance fields. In European Conference on Computer Vision, pages 333-350. Springer, 2022.
[34] Sara Fridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479- 12488, 2023.
[35] Liangchen Song. Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics, 29(5):2732 222222222227 2023.
12

---

[36] Benjamin Attal, Jin-Bin Huang. Christian Richardt, Michael Zollhoefer, Johanmes Kopf, Matthew O O'Toole. and Changil Kim. Hyperreel: High-fidelity 6-dof video with ray-conditioned sampling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 16610-16620, 2023.
[37] Ruijie Zhu, Yanzhe Liang, Hanzhi Chang, Jiacheng Deng. Jiahao Lu, Wenfei Yang. Tianzhu Zhang, and Yongdong Zhang. Motiongs: Exploring explicit motion guidance for deformable 3d gaussian splatting. arXiv preprint a70:v22400007007, 2024.
[38] Yatian Pang, Wenxiao Wang, Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoencoders for point cloud self-supervised learning. In European conference on computer vision, pages 604-621. Springer, 2022.
[39] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu, and Hengshuang Zhao. Point transformer v2: Grouped vec- tor attention and partition-base based pooling. Advances in Neural Information Processing Systems, 35:33330- 33342, 2022.
[40] Weiguang Zhao, Yuyao Yan, Chaolong Yang, Jianan Ye, Xi Yang. and Kaizhu Huang. Divide and conquer: 3d point cloud instance segmentation with point-wise binarization. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 562-571, 2023.
[41] Jiahao Lu, Jiacheng Deng, Chuxin Wang, Jianfeng He, and Tianzhu Zhang. Query refinement transformer for 3d instance segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 185 18516-1526, 2023.
[42] Jiacheng Deng. Jiahao Lu, and Tianzhu Zhang. Unsupervised template-assisted point cloud shape corre- spondence network. arXiv preprint 24X.::2403.14122 2024.
[43] Jeongmin Bae, Seoha Kim. Youngsik Yun, Hahyun Lee, Gun Bang, and Youngjung Uh. Per-gaussian embedding- based deformation for deformable 3d gaussian splatting. arXiv preprint wrXiv:2404.03613 2024.
[44] Johannes I. Schonberger and Jan-Michael Frahm. Structure-from-mte motion revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[45] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 1-9, 2022.
[46] Cheng Sun, Min Sun, and Hwann-Tzone Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469) 2022.
[47] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285 -295, 2023.
[48] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing. 13(4):600-612, 2004.
[49] Richard Zhang, Phillip Isola, Alexci Efros, Eli Shechtman, and Oliver Wang. The unreasonable elfectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 586-595, 2018.
[50] P Kingma Diederik. Adam: A method for stochastic optimization. (No Tule), 2014.
[51] Zeyu Yang. Hongye Yang. Zijie Pan, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting, arXiv preprint arXiv:2310.1666 10642, 2023.
[52] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usumier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-288 2020, Proceedings, Part I 16, pages 213-229. Springer, 2020.
[53] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pages 1290-1299, 2022.
[54] Xiang Guo, Jiadai Sun, Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yumeng Zhang, and Jingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16022-16033, 2023.
13

---

A Appendix supplemental material
A.1
Overview
This supplementary material provides more model and experimental details to understand pro- posed method. After that, we present more experiments to demonstrate the effectiveness of our methods. Finally, show rich visualization of our modules.
A.2 More Model Details
Feature Encoding, As illustrated in Section 4.3.1 we can utilize the MLPs from D3DGS 13 the HexPlanes from 4DGaussian [T] as Feature Encoding. Specifically, for both Plenoptic Video [ and HyperNeRF [2]. we use the HexPlanes to encode per-gaussian's s feature. The complete details can be referred to 4DGaussian's main text. For NeRF-DS [47], we utilize the MLPs for feature encoding. The complete details can be referred to D3DGS's main text.
A.3 More Implementation Details
both Plenoptic Video and HyperNeRF, the total training comprises 14,000 iterations, with the first stage encompassing 5,000 iterations. For NeRF-DS, the total training comprises 40,000 iterations, with the first stage encompassing 15,000 iterations. The dimension of attribute y is set as 16.
For
A.4 Comparison with 3DGStream
Similarities: Our method and 3DGStream both perform deformations on 3D Gaussians (3DGs) where absolute positions and relative positional relationships more accurately maintained. For each timestep i, 3DGStream uses the 3DGs from the previous timestep i as initialization. In contrast, method uses canonical Gaussians (which are iime-independent) the initialization. To achieve accurate relative positional relationships and minimize noise interference from the canonical Gaussians, we employ two-stage deformation strategy. The first stage obtains accurate 3DGs, and the second stage further deforms these 3DGs to achieve preciser rendering results.
Ad Advantages: 1. Flexibility: Unlike 3DGStream,
method does not require the results from the previous timestep. We can render at any arbitrary time without needing the previous timestep's 3DGs. On the other hand, 3DGStream relies the 3DGs from the previous timestep for rendering the next. Robustess: 3DGStream heavily depends on the reconstruction quality at timestep 0. the initial reconstruction is poor, subsequent reconstructions will be negatively affected, and these errors accumulate over time. Our method, however, starts with noisy canonical Gaussians and improves the reconstruction quality through two-stage deformation process, resulting in progressively better reconstructions.
Disadvantages: Training Efficiency: 3DGStream employs online reconstruction approach, leading to shorter training time and faster rendering speed. In contrast, our method involves offline training, which results in relatively longer training time.
A.5 Detailed Results
In T Table and Table g we provide the results for individual scenes associated with Section 5.2 of the main text. It can be observed that our method achieved superior metrics in almost every scene compared to previous methods, demonstrating the effectiveness and generalization of our method under various scenes.
A.6 More Ablation Studies
The effectiveness of V. As depicted in Table conduct ablation study on the flame steak scene of Plenoptic Video dataset to examine the impact of y. From the table, is evident that y can enhance rendering quality and V=16 is the best setting.
Ablation study of timestep in Temporal Aggregation Moudle, From Table 10 is evident that when timestep is set 1 x, the performance is the best. We speculate that the reason for the lowest
14

---

Cut Beef
Cook Spinach
Sear Steak
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
NeRFPlayer 35
31.83
0.928
32.06
0.930
32.31
0.940
HexPlane 15
32.71
0.985
31.86
0.983
32.09
0.986
KPlanes 34
31.82
0,966
32.60
0.966
32.52
0.974
Mix Voxels [3
31.30
0.965
31.65
0.965
31.43
0.971
4DGaussian
32.90
0.957
32.46
0,949
32.49
0.957
Ours
33.49
0.960
32.91
0.951
33.98
0.959
Flame Steak
Flame Salmon
Coffee Martini
Method
PSNR
SSIM
PSNR
SSIM
PSNR
SSIM
NeRFPlayer 35
27.36
0.867
26.14
0.849
32.05
0.938
HexPlane [5
31.92
0.988
29.26
0.980
KPlanes 34
32.39
0.970
30.44
0.953
29.99
0.953
Mix Voxels [31
31.21
0.970
29.92
0.945
29.36
0.946
4DGaussia
32.51
0.954
29.20
0.917
27.34
0.905
Ours
33.51
0.958
29.19
0.921
29.04
0.915
Table 6: Per-scene results PlenopticVideu dataset.
3D Printer
Chicken
Broom
Banana
Method
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
PSNR
MS-SSIM
Nerfies 19
20.6
0.83
26.7
0.94
19.2
0.56
22.4
0.87
HyperNeRF 2
20.0
0.59
26.9
0.94
19.3
0.59
23.3
0.90
TiNeu 45
22.8
0.84
28.3
0.95
21.5
0.69
24.4
0.87
FFDNeRF [54]
22.8
0.84
28.0
0.94
21.9
0.71
24.3
0.86
3D-GS [12
18.3
0.60
19.7
0.70
20.6
0.63
20.4
0.80
4DGaussian m
20.9
0.75
24.1
0.82
20.0
0.53
22.2
0.74
OursI
21.6
0.78
26.2
0.89
20.8
0.57
24.7
0.84
4DGaussian
22.1
0.81
28.7
0.93
22.0
0.70
28.0
0.94
Ours*
22.1
0.81
29.2
0.95
22.3
0.74
28.7
0.95
Table 7: Perscene results of HyperNeRR dataset by different models. Here, represents that we train the model based on Sparse Init. represents that we train the model based on Dense Init.
PSNR
Method
Sieve
Plate
Bell
Press
Cup
As
Basin
3D-GS [2]
23.16
16.14
21.01
22.89
21.71
22.69
18.42
TiNeuVox
21.49
20.58
23.08
24.47
19.71
21.26
20.66
HyperNeRF 2
25.43
18.93
23.06
26.15
24.59
25.58
20.41
NeRF-DS 47
25.78
20.54
23.19
25.72
24.91
25.13
19.96
D3DGS [
25.72
20.40
25.24
25.70
24.35
26.35
19.70
Ours
26.60
20.92
25.48
26.24
24.95
26.54
19.79
Table Perscene results of NeRF-DS dataset by different models.
performance at 0.5x might be attributed the absence of images corresponding to +0.5 timestep in the dataset. Consequently, the lack of supervision at this timestep may induce significant feature biases, resulting in relatively poorer performance.
Ablation study of K in Denoised Spatial Aggregation Moudle, From Table is evident that setting K = 16 yields the best results.
Ablation study of the iteration rounds the first stage. From Table 12 is evident that the two-stage training strategy is meaningful. If we train both deformation operations simultaneously. the performance poor, as indicated in the first row. Only by first training the first deformation network and then proceeding train the second deformation network after the first deformation network
15

---

Setting
Dim
PSNR
SSIM
W/o y
33.13
0.956
W
4
33.33
0.957
16
33.51
0.958
W }
32
33.40
0.958
W y
64
33.29
0.957
Table 9: The effectiveness of y. We compare the results on the flame steak scene of Plenoptic Video dataset.
Setting
PSNR
SSIM
0.5
33,40
0.957
1x
33.51
0.958
2x
33.45
0.957
Table 10: Ablation study of timestep in Temporal Aggregation Moudle. We compare the results on the flame steak scene of PlenoptieVides dataset.
K
PSNR
SSIM
4
33.39
0.957
16
33.51
0.958
32
33.35
0.957
Table 11: Ablation study K in Denoised Spatial Aggregation Moudle. We compare the results on the flame steak scene of Plenoptic Video dataset.
Rounds
PSNR
SSIM
0
32.89
0.951
4000
33.40
0.957
6000
33.38
0.957
8000
33,51
0.958
10000
33.37
0.957
Table 12: Ablation study of the iteration rounds of the first stage. We compare the results on the flame steak scene of Plenoptic Video Video dataset.
well-trained can we achieve optimal performance. Additionally, the optimal number of iteration roudns is 8000.
Ablation study of the iteration rounds of the first stage.
A.7 Assets Availability
The datasets that support the findings of this study available in the following repos- itories: HyperNeRF 2 at hhttps://gtthn com/ under Apache-2.0 license. NeRF-DS [47]
cww/ httpa//u..cooooooo//a////e/eeee
CattteerraanNNRRRF---- DS/
under Apache 2.0 license. Plenoptic [1] at https 3D Video
under CC
BY-NC 4.0 license. The code of our baseline [I] 13 is available at httpp:
com/
ingra 1 41/1mmmmmeeffi ceeeformabee220 3Dw.-aaaaaiammmmm under MIT license and mndhttts://giiee htt.au...cooouuutttttt hustvl/
4DGaussian under Gaussian- Gaussian-Sllatting Splatting license.
16

---

4DGaussian
Ours
GT
Figure 10: Qualitative comparisons Plenoptic Video Dataset.
A.8 More Visual Comparison
Figure 10 shows visual comparisons Plenoptic Video Dataset. We compare the results of 4DGaussian and our model.
Figure 11 and Figure 12 shows more visual comparisons HyperNeRF Dataset. We compare the results of 4DGaussian and our model.
Figure 13 shows more visual comparisons on NeRF-DS Dataset. We compare the results of 4DGaus- sian and our model.
The above visual comparisons demonstrate that our method preserves better rendering quality while containing fewer artifacts.
17

---

4DGaussian
Ours
GT
4DGaussian
Ours
GT
Figure 11: Qualitative comparisons on HyperNeRF Dataset.
NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We accurately reflect the contributions and scope in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
18

---

HyperNeRI
4DGaussian
4DGaussian
D3DGS
D3DGS
GT
-Ours
+Ours
Figure 12: Qualitative comparisons on HyperNeRF Dataset.
It is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Please refer Section 5.4 in the main text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
19

---

4DCaussian
Ours
GT
4DGaussian
Ours
Figure 13: Qualitative comparisons on NeRF-DS Dataset.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: Please refer to Section 3
Guidelines:
The answer NA means that the paper does not include theoretical results.
20

---

All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Section outlines the structure, while Section 5 delves into the specific experimental settings. For additional details, please refer to the supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The data public datasets and are going to open-source our code.
21

---

Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
wwtpggudddssseeee http:////aeaaaeeeeeeeo//e/aaaaaa http:///ooeeeeeieiiiiiiiiaaaaaee colllcc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide the specific experimental settings in Section 5 More details can be acquired in supplemental material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We use the same datasets and evaluation as previous methods, making this fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
22

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide them Section 5.1
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:::////ui....eeeeeeee http://s....cblloocttccccccaaaaa
Question:
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Our work solely intended for academic research purposes.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
23

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We properly credit the related assets and explicitly mention the license and terms of use. Please refer to Section A.7 in the supplemental material.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cww/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "vv1Ci8l13C", "reviewer": "Reviewer_ELfM", "summary": "This paper proposes DN-4DGS, a deformation-based Gaussian splatting technique for dynamic scene rendering. DN-4DGS utilizes a grid-based representation to model the deformation field. It features a two-stage deformation inference pipeline that aggregates temporal and spatial information to reduce noise in the deformation field. These two stages are trained sequentially. Experimental results show that DN-4DGS outperforms other methods in rendering quality.", "questions": "Compared to the 3DGStream method, what are the advantages and disadvantages of DN-4DGS? Adding a discussion of the latest methods can highlight the unique features of this method.", "limitations": "A potential limitation could be the inability to handle scenarios with significant movement.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "The Temporal Aggregation Module and Denoised Spatial Aggregation Module leverage temporal and spatial features to regularize the deformation field. This idea is both sound and novel. Based on the reconstructed canonical Gaussians demonstrated in the paper, I find that the reconstructed deformation field may be closer to the true deformation. The authors provide clear intuitions about these designs.", "weaknesses": "There are some inaccurate description in the paper. In line 158, $G_t$ has the time parameter t. Does this mean that the position of the Gaussian is deformed to the time t? In formulas (3) and (5), the function depends only on the current coordinates x, y, z and time t. However, it actually also involves temporal features and neighboring points in space. These parameters are not described in these two formulas.\n\nThe authors claims that the reason for performing temporal optimization first is to reduce the impact of noise on the KNN in the second stage. However, in the ablation study, there is a lack of experiments that only use DSAM without TAM, where TAM could be replaced with an ordinary deformation network. Therefore, it is difficult to determine the correctness of this claim.\n\n\nThe authors did not provide video results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thank you for pointing that out. It was indeed a typo, and we will correct \"$G_t(i)$\" to \"$G(i)$\".\n2. Thank you for your reminder. We will correct the above formulas and add these two parts.\n    \n    Formula 3: $\\Delta x, \\Delta y, \\Delta z  = \\Psi(x, y, z, T_n)$, where $T_n$ represents the set of $t$'s neighbors.\n    \n    Formula 5: $\\Delta x', \\Delta y', \\Delta z'  = \\Psi'(P_k, t)$, where $P_k$ represents the set of $k$ neighbors of ($x′, y′, z′$).\n    \n3. Thank you for your suggestions. We conducted experiments where we used only DSAM without TAM, replacing TAM with an ordinary deformation network. The results can be seen in the table below. In the camera-ready version, we will incorporate this experiment into the original table to make it complete.\n    \n    \n    | Setting | PSNR  | SSIM |  LPIPS |\n    | --- | --- | --- | --- |\n    | w TAM | 32.02  | 0.944  | 0.043 |\n    | w/o TAM | 31.72   | 0.943 |  0.045 |\n4. Video results: According to NeurIPS 2024 policy, we can only provide the video link to the Area Chair (AC). Please contact the AC for the video link.\n5. **Similarities:** Our method and 3DGStream both perform deformations on 3D Gaussians (3DGs) where absolute positions and relative positional relationships are more accurately maintained. For each timestep $i$, 3DGStream uses the 3DGs from the previous timestep $i-1$ as initialization. In contrast, our method uses canonical Gaussians (which are time-independent) as the initialization. To achieve accurate relative positional relationships and minimize noise interference from the canonical Gaussians, we employ a two-stage deformation strategy. The first stage obtains accurate 3DGs, and the second stage further deforms these 3DGs to achieve preciser rendering results.\n    \n    **Advantages:**\n    \n    1. **Flexibility:** Unlike 3DGStream, our method does not require the results from the previous timestep. We can render at any arbitrary time without needing the previous timestep's 3DGs. On the other hand, 3DGStream relies on the 3DGs from the previous timestep for rendering the next.\n    2. **Robustness:** 3DGStream heavily depends on the reconstruction quality at timestep 0. If the initial reconstruction is poor, subsequent reconstructions will be negatively affected, and these errors can accumulate over time. Our method, however, starts with noisy canonical Gaussians and improves the reconstruction quality through a two-stage deformation process, resulting in progressively better reconstructions.\n    \n    **Disadvantages:**\n    \n    1. **Training Efficiency:** 3DGStream employs an online reconstruction approach, leading to shorter training time and faster rendering speed. In contrast, our method involves offline training, which results in relatively longer training time.\n6. Thank you for your reminder. We will consider the application of our method in scenarios with significant movement.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WWuXxfneZA", "reviewer": "Reviewer_1vtH", "summary": "This paper presents a two-stage pipeline for reconstructing 4D scenes with 3D Gaussians. Similar to previous works, the authors defined a set of canonical 3D gaussians and use MLPs as the deformable fields to deform them into new frames. The key difference is that the authors notice that the deformable field contains a lot of noise, so they propose to suppress the noise by using a two-stage deformation process. In the first stage, they use a temporal aggregation to aggreate temporal information. In the second stage, spatial aggregation is performed with nearest neighbor. The training loss is standard gaussian noise with TV loss. Results demonstrated that the proposed method outperforms previous methods.", "questions": "What is the training time of this method compared with other 4D Gaussians? Would two stage training and aggregation delay training speed?\n\nConsidering the weaknesses I mentioned, I am happy to discuss with the authors and hope to see a better exposition of this paper, especially about the NSS and how that relates to DTS.", "limitations": "The authors have discussed their limitations.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The authors notice that the problem of current 4DGS is that the canonical space contain a lot of noise\n- The authors propose a noise suppression strategy and temporal-spatial aggregation to remove the noise\n- The authors perform extensive experiments and ablation studies to verify their idea", "weaknesses": "Personally I feel like the biggest issue for this paper is the presentation for the introduction and method are not that clear for me. Firstly, the authors claim that previous 4D Gaussian methods can have noise canonical space. I am not fully understand what does noise means here, there should be more explanation on this. Secondly, it seems that the noise supression strategy is just using temporal aggregation followed by spatial aggregation. But the authors claim that there are two components in Line 46: NSS and DTS, but seems that they represent the same thing. On Line 140, the authors claim that after the first deformation, the noise is attenuated. Why is that? With this being said, what is the difference of the last two rows of Table 4?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Weaknesses:** ###\n\nWe apologize for any confusion caused by our expression. Below, we will clarify your questions.\n\n1. **\"Noise\"**: In the canonical + deformable design, we input the canonical Gaussian coordinates $x, y, z$ and time $t$ into the deformable network. The deformable network essentially performs basic operations (addition, subtraction, multiplication, division) on the coordinates $x, y, z$ and time $t$. Since the point-to-point relationships within the canonical Gaussians are chaotic and erroneous, as shown in Figures 2 and 5 of the main text, it is predictable that feeding these erroneous coordinates into the deformable network will transfer this error into the deformation field, introducing inaccuracies in the final deformations $\\Delta x, \\Delta y, \\Delta z$.\n2. **NSS and DTS**: NSS is a **strategy** that uses two stages of deformation to reduce the impact of noise on the deformable network. During this process, we have two training phases. In the early training phase, we only supervise the first deformation. Once the Gaussian coordinates obtained from the first deformation are relatively accurate, we add the second deformation and shift the supervision to it, as detailed in **Figure 1** of the **attached PDF**. This design does have limitations, which can be seen in Section 5.4 of the main text. DTS is a specific **feature aggregation method** we propose. Unlike 4DGaussian, D-3DGS lacks explicit spatiotemporal aggregation, so we designed DTS to aggregate spatiotemporal information. Considering that inaccurate coordinate relationships would be amplified through spatial aggregation (KNN), we only perform spatial aggregation in the second stage of NSS, which is DSAM. We name the spatial aggregation module \"Denoised Spatial Aggregation Module\" (DSAM) because the Gaussian coordinates input into DSAM are more accurate (denoised), as shown in the fourth column of Figure 8 in the main text. Therefore, we prefix the Spatial Aggregation Module with \"Denoised.\" DSAM itself does not have denoising capabilities; it solely performs spatial feature aggregation.\n3. **\"The authors claim that after the first deformation, the noise is attenuated. Why is that?\"**: We emphasize that there are two training phases in the process. In the first phase, we only train the first deformation operation (TAM). Once the Gaussian coordinates obtained from the first deformation are relatively accurate, we add the second deformation (DSAM) into the network for training. Because the Gaussian coordinates and their relationships are relatively accurate after the first phase of training, we claim that after the first deformation, the noise is attenuated.\n4. **Difference Between the Last Two Rows of Table 4.** We apologize for the oversight in not clearly explaining the settings for the penultimate row. The penultimate row indicates that we did not employ two training phases; instead, we trained TAM and DSAM simultaneously throughout the entire process. The main issue with this training approach is that the Gaussian coordinates deformed by TAM are never explicitly supervised, and thus, they still contain noise. Consequently, the Gaussian coordinates input into the DSAM module remain noisy. This is reflected in the quantitative results, where the performance in the penultimate row is actually lower compared to the third row of Table 4.\n### **Questions:** ###\n\n\nWe reported the time in the \"Time\" columns of Table 1 and Table 2 in the main text. Indeed, this two-stage training and aggregation approach does result in some delay in training speed. However, the difference is not substantial, and the method remains competitive. The specific reason is that we reduced the number of MLP layers in the deformable network.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zYBwJ79uWr", "reviewer": "Reviewer_3vN1", "summary": "The author proposed a Denoised Deformable Network to enhance the rendering performance of 4D-GS. In this network, a two-stage deformation prediction method was introduced to suppress noise and achieve better performance.", "questions": "Method: 1. The description of the method is unclear. In section 4.3.1, the feature $F_{t}(i)'$ is obtained by concatenating $F(i)$, $F_{max}(i)$ and $\\gamma_{i}$. However, $F(i)$ and $F_{max}(i)$ have different dimensions. It is not explained how these are concatenated or what the shape of $\\gamma_{i}$ is.\n\n2.During optimization, why is an additional embedding $\\gamma_{i}$ included? What is its purpose? For instance, in HexPlane, pairs of 2D planes are initialized to represent specific spatial or spatiotemporal axes. How does $\\gamma_{i}$ function in this context?\n\n3. In the DSAM (section 4.3.2), the operation is similar to that in section 4.3.1. There is no direct denoising operation, only a two-stage deformation prediction.\n\nRelated Work and Comparison: Many new works have appeared in CVPR 2024 in terms of dynamic NeRF and Gaussian diffusion.\n\nAblation study: From the methods section, it appears that the NSS includes both the TAM and DSAM. How, then, can it be part of the ablation study?", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Based on the performance on three dynamic datasets, the proposed method outperforms the compared methods.", "weaknesses": "1. Novelty: The author claims that the novelty of the proposed method lies in suppressing noise to achieve more accurate deformation prediction. However, from the figures and descriptions, it appears to only apply a two-stage deformation prediction without any denoising operations.\n\n2. Lack of detailed explanation of the method.\n\n3. Insufficient comparison with other methods and related work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **1. Novelty** ###\n\nThe primary motivations of this paper are twofold:\n\n1. **Noise in Canonical + Deformable Design**: In the canonical + deformable design, point-to-point relationships within the canonical Gaussians are chaotic and erroneous (noisy), which can be transferred into the deformation field. To address this issue, we propose the **Noise Suppression Strategy (NSS)**. This strategy involves two stages of deformation to mitigate the impact of noise on the deformation network. During the first stage, we supervise only the initial deformation. Once the Gaussian coordinates from the first deformation are relatively accurate, we introduce the second deformation and shift the supervision to it. The rendering quality can be progressively improved through this approach.\n2. **Lack of Explicit Temporal-Spatial Feature Aggregation**: Previous methods lacked explicit aggregation of temporal and spatial features. We address this by proposing the **Decoupled Temporal-Spatial Aggregation Module (DTS)**, which includes:\n    - **Temporal Aggregation Module (TAM)**: Handles the temporal aggregation process.\n    - **Denoised Spatial Aggregation Module (DSAM)**: Manages spatial aggregation.\n    \n    Given that inaccuracies in coordinate relationships can be amplified  during spatial aggregation (KNN), we decouple temporal and spatial aggregation, placing them in two distinct deformation stages. Since the Gaussian coordinates obtained after the first deformation are relatively accurate, we perform spatial aggregation in the second stage to avoid the amplification of noise due to inaccurate spatial relationships. We name this module **Denoised Spatial Aggregation Module (DSAM)** because the Gaussian coordinates input into DSAM are more accurate (denoised), as shown in the fourth column of Figure 8 in the main text. The term \"Denoised\" is used to highlight the improved accuracy of the inputs rather than implying that DSAM performs denoising.\n    \n### **2. Method** ###\n\n1. **Concatenation in the Formula**: In this formula, we concatenate $F_t(i)\\in \\mathbb{R}^{ 1\\times C_1}$ with $F_{\\text{max}}(i) \\in \\mathbb{R}^{ 1\\times C_2}$ and $\\mathcal{Y}_i \\in \\mathbb{R}^{ 1\\times C_3}$, rather than $F(i)$. We concatenate along the second dimension to obtain $F_t(i)' \\in \\mathbb{R}^{ 1\\times (C_1+C_2+C_3)}$.\n2. **Purpose of** $\\mathcal{Y}_i$: The primary purpose of $\\mathcal{Y}_i$ is to add a coordinate-independent embedding to improve quality by accurately modeling different deformations of various Gaussians without adjacency interference. We have also observed a similar design in a recent work, **Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting (E-D3DGS) [1]**, presented at ECCV 2024.\n3. **Clarification on DSAM**: We apologize for any misunderstanding caused by our previous description. In fact, **DSAM itself does not perform denoising but is focused on spatial feature aggregation**. The name **Denoised Spatial Aggregation Module (DSAM)** was chosen because the Gaussian coordinates entering DSAM are more accurate (denoised) after the first deformation, as illustrated in the fourth column of Figure 8  in the main text. Hence, we prepend \"Denoised\" to emphasize the improved accuracy of the inputs rather than suggesting that DSAM performs denoising.\n\n### **3. Related Work and Comparison** ###\n\nThank you for your suggestions. We will incorporate references to and discussions of the following papers in the related work section. Additionally, we have compared these approaches using the PlenopticVideo dataset.\n\n| Method | PSNR  | LPIPS |\n| --- | --- | --- |\n| 4DGS[2] | 32.01  | **0.014** |\n| E-D3DGS[1] | 31.42 | 0.037 |\n| 4DGaussian[3] | 31.15   | 0.049 |\n| 4DGaussian+ours | 32.02  | 0.043 |\n| spacetime-GS[4] | 32.05 | 0.026 |\n| spacetime-GS+ours | **32.42**  | 0.023 |\n\n### **4. Ablation study** ###\n\nHere's a detailed explanation of the ablation study for Table 4:\n\n- **The second row** refers to using two-stage deformation operations, but both stages of feature extraction networks still utilize the ordinary deformation network.\n- **The third row** indicates that we do not perform two-stage deformation operations. Instead, we directly replace the ordinary deformation network from the first row (baseline) with our proposed TAM.\n- **The fourth row** means that we replace the feature extraction network used in the first stage of the two-stage deformation operations (from the second row) with our proposed TAM.\n- **The fifth row** specifies that we do not use two-stage training. Instead, we couple temporal and spatial aggregation into a single process, training TAM and DSAM simultaneously throughout the entire process.\n- **The last row** shows that we replace the feature extraction network used in the second stage of the two-stage deformation operations (from the fourth row) with our proposed DSAM.\n\n[1] Bae J, Kim S, Yun Y, et al. Per-Gaussian Embedding-Based Deformation for Deformable 3D Gaussian Splatting[J]. arXiv preprint arXiv:2404.03613, 2024.\n\n[2] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642.\n\n[3] Wu G, Yi T, Fang J, et al. 4d gaussian splatting for real-time dynamic scene rendering[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 20310-20320.\n\n[4] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2023. Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis. arXiv preprint arXiv:2312.16812 (2023).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SnRoEm1yIC", "reviewer": "Reviewer_umjH", "summary": "This paper proposes a novel framework for modeling dynamic scenes. To address the noisy 3D Gaussians in canonical space, the authors introduce a Noise Suppression Strategy (NSS) with a two-layer Deformation Field to resolve this issue. Additionally, they propose a Decoupled Temporal-Spatial Aggregation Module (DTS) to efficiently aggregate neighboring temporal features (for the first deformation) and k-nearest neighbors' spatial features (for the second deformation). Experiments on real-world datasets show substantial quality improvements compared to D-4DGS [1].\n\n\n\n[1] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.", "questions": "A minor question: I would like to know if the authors have tested the effectiveness of their method on the D-NeRF dataset.", "limitations": "Please refer to the weakness.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper aims to improve the capability of 3DGS-based dynamic reconstruction methods in modeling real-world scenes. The proposed Temporal Aggregation and Spatial Aggregation are sensible, and the detailed ablation studies in Tab. 4 demonstrate the effectiveness of each component. In terms of rendering metrics, the experimental results in this paper surpass the previous state-of-the-art 3D-GS-based methods in real-world scenes.\n\nThe design of the dual-deformation network and the experiments with strong baselines are appreciated.", "weaknesses": "1. I think the following papers should also be cited:\n   - (CVPR 2024) Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al.\n   - (ICLR 2024) Real-time Photorealistic Dynamic Scene Representation and Rendering with 4D Gaussian Splatting by Zeyu Yang et al.\n2. Issues with the HyperNeRF dataset: I believe that the visualization comparisons on the HyperNeRF dataset should include HyperNeRF [2], 4DGS [3], and even Deformable-GS [4]. Based on my understanding of the HyperNeRF dataset, its camera poses are not accurate. This leads to a slight misalignment between the rendered images and the ground truth due to the inaccurate camera poses. Most current image quality evaluation metrics, such as PSNR and SSIM, heavily penalize slight misalignments more than blurriness. For example, in the Fig. 7 `Chicken` scene, I found that both D-4DGS [1] and DN-4DGS exhibit blurriness under the sparse Init setting. I am confident that HyperNeRF and Deformable-GS can render relatively sharp images, but slight misalignments with the ground truth caused by inaccurate poses result in these methods having much lower rendering metrics compared to some blurrier methods. (In Tab. 2, HyperNeRF, which has higher rendering quality than D-4DGS, has lower rendering metrics.) Therefore, I will not rely heavily on the quantitative metrics of the HyperNeRF dataset. The visual results are more important, and comparisons with more previous state-of-the-art methods are needed to demonstrate the effectiveness of the proposed method.\n3. Selection of the baseline. I do not think D-4DGS [1] is an appropriate baseline. The essence of why the Deformation Field (outputting delta) works is due to its sufficiently smooth output. D-4DGS's Hexplane clearly disrupts this smoothness, which is why it performs poorly on the D-NeRF dataset. In real-world scenes, due to the high sensitivity of 3D Gaussian explicit representation to inaccurate camera poses, it does not have a significant advantage over NeRF-based methods (like HyperNeRF). I fully agree with the authors' attempt to improve the modeling capability of Deformation-based Gaussian splatting, but I think the authors need to explore more fundamental reasons, such as the accuracy of camera poses and the negative optimization effects of D-4DGS's strong regularization of deformation in real-world scenarios. Methods like 4DGS [3] and spacetime-GS [5], which decouple time features into each GS, can better handle the inaccuracies of camera poses in real-world scenes\n\n\n\n[2] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228, 2021.\n\n[3] Zeyu Yang, Hongye Yang, Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4d gaussian splatting. arXiv preprint arXiv:2310.10642.\n\n[4] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023.\n\n[5] Zhan Li, Zhang Chen, Zhong Li, and Yi Xu. 2023. Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis. arXiv preprint arXiv:2312.16812 (2023).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "1. Thank you for your suggestions. We will cite the relevant works as recommended.\n2. We appreciate your understanding of HyperNeRF and the insights regarding both HyperNeRF and Deformable-GS. In response, we have conducted additional visual comparisons in **Figure 2** of the **attached PDF**, including HyperNeRF and Deformable-GS. It is worth emphasizing that our method is also applicable to Deformable-GS. As shown in **Figure 3 (b)** of the **attached PDF**, while the canonical GS in Deformable-GS performs better than D-4DGS, it still contains noise. Consequently, we not only present results for D-4DGS + our method but also for Deformable-GS + our method. The visual results demonstrate that our approach provides notable improvements.\n3. Thank you for your suggestions. In fact,  our goal is to design our method as a plug-and-play solution. Therefore, we have conducted experiments using both D-4DGS and Deformable-GS as baselines. Specifically, Tables 1 and 2 in the main text present results with D-4DGS as the baseline, while Table 3 in the main text includes results with Deformable-GS as the baseline. To provide further insights, we have also shown results for Deformable-GS + our method on the HyperNeRF dataset in **Figure 2** of the **attached PDF**. Additionally, we present results for SpaceTime-GS + our method on the PlenopticVideo dataset in the table below, highlighting the effectiveness of our two-stage deformation strategy (NSS) and spatial aggregation process (DSAM). \n    \n    \n    | Method | PSNR | LPIPS |\n    | --- | --- | --- |\n    | D-4DGS | 31.15 | 0.049 |\n    | D-4DGS+ours | 32.02 | 0.043 |\n    | spacetime-GS | 32.05  | 0.026 |\n    | spacetime-GS+ours | **32.42**  | **0.023** |\n    \n    We appreciate your suggestions on exploring new directions and acknowledge the importance of camera pose accuracy for deformation-based Gaussian splatting. We also recognize the potential negative impacts of strong deformation regularization in D-4DGS. As a result, we are also actively investigating methods to optimize camera poses through network feedback and exploring new ways to better represent deformation fields.\n    \n4. Our work initially focused on improving dynamic reconstruction in real-world scenarios, which is why we did not extensively test on the synthetic D-NeRF dataset. However, we appreciate your reminder, as testing on D-NeRF is indeed valuable for demonstrating the generalization and completeness of our method. Consequently, we have conducted experiments on D-NeRF as well. As illustrated in the **Figure 3 (a)** of the **attached PDF**, we have visualized the canonical results for both D-4DGS and Deformable-GS. The results show that Deformable-GS has less noise compared to D-4DGS, but noise is still present in moving areas. The quantitative results on D-NeRF dataset, as shown in the table below, indicate that our method improves performance on both D-4DGS and Deformable-GS, with enhancements on D-4DGS surpassing those on Deformable-GS.\n| Method | PSNR | SSIM |\n| --- | --- | --- |\n| D-4DGS | 34.05  | 0.9787 |\n| D-4DGS+ours | 34.53  | 0.9811 |\n| Deformable-GS | 39.51 | 0.9902 |\n| Deformable-GS+ours | **39.87** | **0.9922**  |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Thank you to the reviewers for their valuable reviews, which will help us further improve the quality of our work. We have addressed each reviewer's questions individually within their respective sections, and we look forward to further discussions during the discussion phase. Below and in the attached PDF, we present more comprehensive understanding and visualizations of our method.

Meta-review: This submission received all positive reviews after a strong rebuttal. The reviewers feel the proposed method is sound and novel, and the ablation study is mostly solid. The AC recommends accepting the paper, but asks the authors to carefully address the concerns raised by the reviewers in the camera-ready version, including the clarity of the exposition, references suggested by the reviewers, and additional ablations.
Decision: Accept (poster)