Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting
Authors: Jiawei Xu
Zexin Fan
Jian Yang
Jin Xie
Venue: NeurIPS 2024 poster (2024)
Keywords: Dynamic Scene Rendering
Gaussian Splatting
Hash Encoding
Explicit Representation
Primary area: machine_vision
TL;DR: This paper proposes Grid4D, a novel high-fidelity dynamic scene rendering model with 4D decomposed hash encoding.

Abstract: Recently, Gaussian splatting has received more and more attention in the field of static scene rendering. Due to the low computational overhead and inherent flexibility of explicit representations, plane-based explicit methods are popular ways to predict deformations for Gaussian-based dynamic scene rendering models. However, plane-based methods rely on the inappropriate low-rank assumption and excessively decompose the space-time 4D encoding, resulting in overmuch feature overlap and unsatisfactory rendering quality. To tackle these problems, we propose Grid4D, a dynamic scene rendering model based on Gaussian splatting and employing a novel explicit encoding method for the 4D input through the hash encoding. Different from plane-based explicit representations, we decompose the 4D encoding into one spatial and three temporal 3D hash encodings without the low-rank assumption. Additionally, we design a novel attention module that generates the attention scores in a directional range to aggregate the spatial and temporal features. The directional attention enables Grid4D to more accurately fit the diverse deformations across distinct scene components based on the spatial encoded features. Moreover, to mitigate the inherent lack of smoothness in explicit representation methods, we introduce a smooth regularization term that keeps our model from the chaos of deformation prediction. Our experiments demonstrate that Grid4D significantly outperforms the state-of-the-art models in visual quality and rendering speed.

Grid4D: 4D Decomposed Hash Encoding for High-Fidelity Dynamic Gaussian Splatting
Jiawei Xu Zexin Fan Jian Yang 8 Jin Xie
IPCA Lab, VCIP, College of Computer Science, Nankai University
2 State Key Laboratory for Novel Software Technology. Nanjing University, Nanjing, China
3 School of Intelligence Science and Technology, Nanjing University, Suzhou, China (jiaweixu, zexin fan) @ mmall.nankai.edu.co &csjxie@ nju.edu.cn ssjyang @nankai.edu.co
Abstract
Recently, Gaussian splatting has received more and more attention in the field static scene rendering. Due to the low computational overhead and inherent flexibility of explicit representations, plane-based explicit methods popular ways to predict deformations for Gaussian-based dynamic scene rendering models. However, plane- based methods rely the inappropriate low-rank assumption and excessively decompose the space-time 4D encoding, resulting in overmuch feature overlap and unsatisfactory rendering quality. To tackle these problems, we propose Grid4D, a dynamic scene rendering model based on Gaussian splatting and employ- ing novel explicit encoding method for the 4D input through the hash encoding. Different from plane-based explicit representations, we decompose the 4D encod- ing into one spatial and three temporal 3D hash encodings without the low-rank assumption. Additionally, we design novel attention module that generates the attention scores directional range to aggregate the spatial and temporal features. The directional attention enables Grid4D to more accurately fit the diverse defor- mations across distinct scene components based on the spatial encoded features. Moreover, to mitigate the inherent lack of smoothness in explicit representation methods, we introduce a smooth regularization term that keeps our model from the chaos of deformation prediction. Our experiments demonstrate that Grid4D significantly outperforms the state-of-the-art models visual quality and rendering speed. Project page:
gi Grid4D-web/
Introduction
Dynamic rendering aims to construct dynamic scenes from images with specific camera poses and timestamps, allowing rendering from arbitrary viewpoints and moments. Traditional methods Neural Radiance Field (NeRF) 266 and deformation fields reconstruct dynamic scenes for arbitrary rendering. However, these works rely on predicting deformations with the over-smooth
full Multilayer Perceptron (MLP) [31 40 48 [0 [9 28 55
47
16 5 18 38 23 45| 21 4 53).
resulting in slow training speeds and artifacts in rendering quality. To address these challenges, explicit representations such as planes
and hash encoding [27] have been introduced enhance
the rendering of dynamic scenes 33 intermediate features generated by the partial forward propagation process grid-like format. This approach allows us to obtain intermediate features by directly interpolating the cached features based on the input, bypassing the need for the full forward propagation process. addition to reducing computing resource consumption, the inherent flexibility of explicit representation offers advantages in rendering more complex scenes.
46, 8
35 36). The explicit representations store the
Cooresponding authors.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Lighthoiit Artitrey Time Revekeegg
4D Docompood Fbsh Emxxdaa
Spare Time Input
xyz
xyt
4D-GS
DefomGS
GridDD (Ours)
xyzt yzt
xzt
Figure I: We propose novel explicit representation method for dynamic scene rendering that decom-
poses the space-time 4D encoding into the 3D format without the unsuitable low-rank assumption. We achieve significant improvements over the state-of-the art models [4 50 in rendering quality.
Recently, Gaussian splatting 13 achieved fast and nigh-fidelity rendering of static scenes. Addi- tionally, many works have employed Gaussian splatting for dynamic scene rendering by deforming
Gaussians based on the timestamp [44[55) 22 77 77 49 6 20 24 12 52 37). Deforming
Gaussians through pre-defined functions is an effective way to reconstruct dynamic scenes with
sufficient viewpoints [22 177. Additionally, implicit and explicit neural networks are more popular for deforming Gaussians in general cases 44 [5 [2. However, fully ALP-based implicit neural networks have limited learning capacity because of their over-smooth inherent property, thereby strug- gling to render several complex scenes and details effectively. Hence, explicit representation might be an available method to address these problems. Prior works such as 4D-GS [44] use plane- based explicit representations to predict Gaussian deformations, decomposing the 4D space-time encoding into a format comprising six 2D planes, but the performance remains unsatisfactory. We consider that the plane-based methods for Gaussian deformation prediction are based the low-rank assumption which assumes that the features for the deformations have a great deal of commonality and could be factorized into a very low-rank format 33 9 44 As shown in Figure 2 when facing Gaussians with massive overlapping coordinates, the over-decompost pveoppecompooition makes the features have excessive overlap which limits their discriminability for deformation prediction. Therefore, such overlap might block the model from predicting different deformations, resulting in low rendering quality.
To address these problems, present Grid4D, novel model with high dynamic scene rendering quality based on Gaussian splatting [13.. Our approach leverages hash encoding (27) and proposes a new explicit representation method. Unlike the plane-based explicit representations relying on the unsuitable low-rank assumption, as shown in Figure [ we decompose the 4D encoding into one spatial 3D hash encoding and three temporal 3D hash encodings. Figure Dillustrates our proposed 4D decomposed hash encoding reduces overlap arising from the over-decomposed plane-based methods, resulting in more discriminative features. Notably, our encoder generates two types of features: spatial features, representing static information across the timeline, and temporal features, capturing dynamic information. For aggregation, we design a novel attention mechanism, directional attention, which leverages spatial features to generate attention scores in a directional range. This directional attention aligns with the observation that deformation consistency within each scene component often varies across different components, and the attention from the spatial features could better help the model fit such differences. However, like other explicit representation models, Grid4D often lacks smoothness. To address this issue, we propose novel training strategy incorporating smooth regularization which mitigates chaotic deformation predictions to enhance rendering clarity.
We compare Grid4D with several state-of-the-art dynamic scene rendering models. Figure [ and the experimental results show that Grid4D outperforms other models significantly in both visual quality and rendering speed. In general, the contribution of this paper can be summarized as the following.
We propose novel explicit representation method for dynamic scene rendering. By de- composing the 4D encoding into four 3D encodings, our 4D decomposed hash encoder effectively represents the features without relying on the low-rank assumption.
We design novel attention module for spatial and temporal feature aggregation. The directional attention module aligns with the variations in deformation consistency across different scene components, thereby enhancing deformation prediction accuracy.

---

We employ a smooth training strategy to ensure the smoothness of our model. The smooth regularization effectively mitigates chaotic deformation predictions, resulting in high clarity in the rendered images produced by Grid4D.
2 Related Works
NeRF-Based Dynamic Scene Rendering. NeRF 266
reconstructs light fields of static scenes by implicit representations and achieves significant visual improvements. To extend NeRF capabilities for reconstructing dynamic scenes, applying implicit deformation fields to static models finds widespread
use in dynamic scene rendering [31. To model dynamic scenes more accurately, various studies
segment scene into components with different attributes for different modeling [10 [38. Moreover,
several works apply higher-dimensional latent codes for the network input [16 28 and incorporate
additional supervision such as flow supervision across frames [40] [
21
5 18
19 and
motion mask supervision 44 Meanwhile, focusing modeling rigid objects is important in improving accuracy because of their unique physical properties and prevalence most scenes 38 53
Additionally, some research addresses the problems of dynamic scene models in several challenging scenes such as dynamic human modeling (.. specular objects [47] and the scenes without camera poses [23]. However, implicit representations based on full MLPs suffer from the over- weer-soothhng inherent property and require time-consuming training processes. On the other hand, explicit representations, such Triplanes (3] and Hash Encoding (27), enhance NeRF by improving both visual quality and training speed. A popular technique for plane-based explicit representations in dynamic scene rendering decomposing 4D inputs into six 2D inputs 2 33 46 33). Also, hash
encoding and 3D grid explicit representations can assist MLPs in predicting deformations with faster
speed and higher precision 8
42 29 36
Gaussian- Based Dynamic Scene Rendering. Recently, Gaussian splatting [3] models static scenes by Gaussian points, achieving both fast training and high visual quality. When it comes to dynamic scene rendering, using 4D Gaussians or deforming Gaussians with pre-defined functions perform well in the cases with sufficient viewpoints [49 6 22
25 Alternatively, deforming the attributes of 3D Gaussians according to timestamps with neural networks has led to better outcomes in general dynamic scene rendering [50] 44
17
37). Fully MLP-based deformation fields achieve high rendering quality [50] but suffer from the over-smooth inherent property, resulting in the failure of some detail rendering and complex scenes. Explicit representation models, for example, 4D-GS [44]. utilize the planes-based methods as the deformation field. Although plane- based representations are more flexible, they are based on the unsuitable low-rank assumption, leading to massive feature overlap and rendering artifacts. Our work mainly focuses on tackling the unsuitable low-rank assumption inherent in plane-based explicit representations to improve the rendering quality of Gaussian-base based models.
15
24
22
52
3 Method
3.1 Prelimaries: Gaussian Splatting
Gaussian splatting [3] is a static scene rendering model, known for its high training speed and visual quality. This model assumes that the scene composed of 3D Gaussian kernels with (1t, S, R, a, c). corresponding to the position, scaling, rotation, opacity, and color. Notably, the color attribute is defined by the spherical harmonic coefficients (SH). To render the scene, by using view transform matrix W and a projective Jacobian matrix J, Gaussians can be splatted onto camera planes [56] 51]
M' JWEWT J² Σ RSS n(
(I)
where E' is the covariance matrix camera planes and Σ is the original Gaussian covariance which can be calculated by the scaling and rotation attributes. Finally, supposing that the pixel on the camera planes p. the splatted Gaussians can be rendered by the volume rendering equation.
N
i-1
C(p)
Σ
c,a:
I
a =ae- -
(2)
where 12 is the projected coordinates of the 3D Gaussians, and N is the number of overlapped Gaussians on the pixel.
3

---

Be (0 53
plane-bosed repliciteepmtseellllliee reprusenlatiio (4D-GS)
x
12 vt ø overtap
encode
yz
2t
decode
decomposed hash encoding (Ours)
Janfom
ACru fo)
XY2 xyt
xat
yat
Xyz
xyt
X2t
y2t
xod
4D-GS
O'urs
feotures with overtap
(a) comparison of encoding overlapping coordinates
(b) t-SNE and deformation visualization
Figure 2: Comparison of our proposed 4D decomposed hash encoding with the plane-based explicit representation [44]. (a) Compared to the plane-based methods based on the low-rank assumption, our methods reduce the overlap ratio in the features from half to a quarter when encoding points A and B with heavily overlapping coordinates. (b) is the t-SNE [39] visualization of all the features, and the colors denote the corresponding represented deformations. The diversity of colors demonstrates that the reduced overlap makes the features represent different deformations more effecttively
In optimization, adaptive density control is crucial for convergence. It involves pruning low-opacity Gaussians and densifying them based the gradients and scaling. However, original Gaussian splatting cannot represent dynamic scenes and needs the help of deformation fields.
3.2 4D Decomposed Hash Encoding
Dynamic scene rendering involves deforming Gaussians according a 4D coordinate (Ir, y, 2, input, where represents the timestamp and (x. y. 2) means the position of a Gaussian. Instead of employing the over-smooth fully MLP-based implicit representations,
use explicit representation for Grid4D. However, existing plane-based explicit representation relies on the unsuitable low-rank assumption which overly decomposes the (I. M/ 2, encoding into (ar, y), (y, =); (1, 2), (ar, f), (y. (z, () plane encodings 33 9 2 [44). As shown in Figure aa.), for instance, considering the Gaussians and with the same y and z coordinates. The plane-based method has the same encoded features in the (y, 2), (y, t), (z, t) planes, Such high overlap ratio might lead to the low discriminability of the features and block the model from fitting different deformations accurately. To address this problem, directly removing the decomposition by simply adding the time dimension to the traditional 3D grid possible way. However, the 4D hyper-grit grii hash encoding
for the 4D hyper-grid hash encoding leads to high collision rates due to the high space complexity O(n²) ) of the 4D hyper-grid [4] Therefore, thoroughly eliminating the overlap might be an available solution.
Tri-axial 4D Decomposed Grid. To address this problem, we propose a novel decompo- sition approach that decomposes the 4D encoding (I., y, 2, into four 3D hash encodings (z, y, 2), (I, y, t), (y, 2, t), (z, z, t). The decomposition allows us to work with fewer parameters, which reduces the space complexity from O(n)) to O(r²) without relying on the low-rank assump- tion. As shown in Figure ((a), the tri-axial decomposition can effectively reduce the overlap ratio from half to quarter, thereby enhaneing each feature to represent the corresponding deformation. Figure (bb) demonstrates that the features encoded by our methods are more discriminative for deformation prediction than plane-based methods.
Multiresolution Hash Encoding, In the original hash encoding technique 277, the grid employed in the encoder has the same resolution across all dimensions. Consistent resolutions could be suitable for static scene rendering, where the isotropic sampling assumption holds in the 3D space. Nevertheless, the sampling of the 4D space is usually anisotropic. which usually sparse in the time dimension. Therefore, in our implementation, the temporal 3D encodings (I, t), (y, z, f), (I, 2, t) have different resolutions in the dimension account for this sparsity. Following the InstantNGP (27). we set the multiple resolutions of each dimension in a geometric progression:
N Nmax N Nman
N₁ = Nmin bJ, b = exp
(3)
L-1
where Nmin, Nmaz the coarsest and finest resolutions, is level number, the max level, and Nj is the resolution select. The grid voxel positions for the input could be calculated by rounding

---

Spanil fediin
xyz: Position of Caussians
Spatial MLP
I: Timeslamp
R₂ Ta,,arrr 4s Deformation of Gaussians
Dot Product Operation
Attentionn
Canonical Gaussians
HYZE
Deformed Gaussians
xyz
Thmpso Redaaw
Temporal MLP
Multi head D eecoder
Resterization
Directional Attention
R Tv, ár, As
4D Decomposed Hash Encoding
Figure 3: The overview of Grid4D. Given the canonical Gaussians and the timestamp, first encode the decomposed input separately. Then we apply the directional attention scores generated by the spatial static features to the temporal dynamic features, and decode the features with tiny multi-head MLP. Finally, the Gaussians deformed by the predicted deformations are splatted by the differentiable rasterization operation [13 render the images for supervision.
down and up in each level [x,] |x NrJ, x = x M1). The voxels in each level could be obtained from the hash table by hashing the corresponding positions:
hx(xi)
=
mod Ti
i= 1, EX!
where ① the bit-wise XOR operation, d is the dimension of the input, क. are unique large prime numbers, Ti is the size of the level 7 hash table. Then the encoded features could be calculated by the trilinear interpolation (3r, y/, z, i) include the spatial and temporal features from the spatial grid hash encoder Gays and temporal grid hash encoders Gayt Gyst: Gazt respectively.
the grid voxel values. Generally, the encoded features of the 4D input
3.3 Multi-head Directional Attention Decoder
Our 4D decomposed hash encoding generates two types of features: temporal features and spatial features. The temporal features represent the information related to the timestamp while the spatial features represent the common information across the timeline. The Gaussians representing different scene components often have various deformations in almost every timestamp. Therefore, the spatial features could be used to help the model fit such variations, and we design the directional attention module for the spatial and temporal feature aggregation.
Directional Attention. We infer the attention features from the spatial grid hash encoder Gzys with a tiny spatial MI.P sax and generate the score a through the following formula,
2 (brys)) 1, bays fa Gayz (x, y,
where & the Sigmoid function. consider that several components probably have entirely opposite deformations against the neighboring Gaussians. For example, the Gaussians for the shadows often have opposite motions relative to the objects. Therefore, different from the common range (0, 1) of the attention score scale it to directional range (-1, 1) to represent neighboring deformations with opposite directions, thereby enhancing the representation ability the attention mechanism.
Then apply the attention score to the activated deformation features encoded by the three temporal grid hash encoders Gryt: Gyzt, Gazt and tiny temporal MLP ft-
h ② (e(Grrg (x. y, !), Gyze (v, z, l), (x,
where @ the dot product operation. Finally, we get the deformation features h with high represen- tation ability. Our experiments demonstrate that our attention module outperforms the architecture
5

---

which either directly decodes the concatenation of the spatial and temporal features or uses the common range (0, 1) of the attention score.
Multi-head Deformation Decoder. The decoder is required to decode the features h to get the Gaussian deformation. Different from the prior works 4 [50],
tiny multi-head MLP D to decode the features and predict the position deformation with a rotation matrix Rg and a translation matrix T. as [2]. Finally, we deform the position, scaling, and rotation of the Gaussians in the canonical space with the predicted deformation.
n' = R4 + Ta, 5 As, R' R+ D(h) = (Rz, Tax Ar, As),
(7)
We define the rotation matrix R. with quaternion for more accurate interpolation and stable optimization. Following Gaussian splatting [3]. the deformed Gaussians could be rendered into images of specific timestamps via differentiable rasterization.
3.4 Training with Smooth Regularization
Although the proposed model architecture could effectively predict the Gaussian deformation, the decomposed hash encoder still suffers from the lack of smoothness, common challenge in most explicit representation methods. We consider that the MLP decoder has the smooth inherent property and does not require additional smoothing. Therefore, without involving the MLP decoder inference for higher efficiency. Generally, to regularize the hash encoder, we propose novel smooth regularization loss.
set our regularization in the feature space
f, ||G (z, y: z, Gzyzt( + Ex, + Ey. Eg, ez)1
(8)
where (Cx, Cy. C2y is the small random perturbation for the input (æ, y, 2, respectively, and Gzyzt is the concatenation of four grid hash encoders. This regularization enforces similarity among encoded features in neighboring regions, thereby making the nearby Gaussians have similar deformations. Due to the difference of spatial and temporal encoding, we use a different regularization setting for the spatial encoding for several cases. Notably, to improve the efficiency, we randomly select partial Gaussians for the regularization instead of using them all. Our experiments demonstrate that this smooth regularization effectively mitigates the deformation chaos, leading to significantly improved rendering clarity.
In general, similar to Gaussian splatting 13 our total loss function can be summarized as the weighted sum of L1 color loss, D-SSIM loss, and the proposed smooth regularization term.
L= (1- Ae) + XcCD-SSIM + ArCr
99)
where Ac A, are the hyperparameters to balance the losses. Following (50] we use the detached Gaussian positions for deformation prediction, which results in better performance. Also, similar to prior works [50 4 we initialize the static canonical Gaussians without deformation the beginning of the training process. Specifically for SfM 332 initialized Gaussians, we shorten or remove the static initialization process. We apply the same adaptive density controller and opacity resetting
mechanism as Gaussian splatting [13]. The pipeline of Grid4D is illustrated by Figure
4 Experiments
In this section, we introduce our experiments conducted on a single RTX 3090 GPU. build our code mainly on PyTorch [30], while we implement our 4D decomposed hash encoder with CUDA/C++. More experimental results and analysis can be found in the supplementary.
4.1 Experimental Setup
Datasets. We evaluate Grid4D on two popular datasets. D-NeRF 31 dataset is public monocular
synthetic dataset that provides accurate and time-vare time-narying camera poses. HyperNeRF (28) dataset is public real-world dataset captured by one or two moving cameras. Neu3D [16] dataset is public dataset captured by multiple cameras with fixed poses. However, different from synthetic datasets, the camera poses of the HyperNeRF and Neu3D datasets are estimated by COLMAP (32 which not accurate. We set the rendering resolutions of the D-NeRF, HyperNeRF and Neu3D datasets to 800 x 800, 536 x 900 and 1352 x 1024 respectively. Notably, find several mistakes in the

---

Hell Wartior
Hook
(uuggnng Jacks
Lego
TiNeuVox
4D-GS
DeformSS
Grid4D (Ours)
Ground Truth
Figure 4: Qualitative comparisons on the synthetic D-NeRF dataset
31 with our baselines
ground truth of the 'Lego' scene in the D-NeRF dataset, as shown in the last row of Figure 4 so we ignore this scene in all quantitative comparisons rendering quality.
50 12]. HexPlane 22] and TiNeu Vox 3 are NeRF-based dynamic scene rendering models, utilizing plane-based and
Baselines. We compare Grid4D with several state-of-the-arr models 2
3D grid explicit representations respectively. 4D-GS 44] and DeformGS (50] are Gaussian-
models, employing plane-based explicit representation and fully MLP-based implicit representation for the deformation fields respectively. SC-GS [12) is model built on DeformGS [50), and proposes
use sparse control points for better dynamic scene rendering and edit.
Hyperparameters. For all datasets, we configure the resolution of the spatial grid hash encoder to span from 16 to 2048 across 16 levels. Meanwhile, the max level number of temporal grid hash encoders remains consistent at 32. We set X and X, to 0.2 and 0.5 for common scenes and follow similar learning rate schedule DeformGS [50
4.2 Comparisons
Comparison of Visual Quality, We compare Grid4D with the sfte-oofthe-ar models on the synthetic D-NeRF 31 dataset (Table I and Figure 4. the real-world HyperNeRF (28) dataset (Table and
Figure 5 and real-world Neu3D Neu333
16 dataset (Table 3 and Figure (). The PSNR, SSIM [33
LPIPS 54 (VGG (4), and MS-SSIM are the metrics denoting visual quality. Notably, the Defor-
mGS 150 model fails to construct several HyperNeRF scenes with large motions and imprecise camera poses, mentioned in their paper. Several failed cases be found in Section B of the supplementary, and we consider that this is also due to the over-smooth inherent property of fully MLP-based implicit representation.
Due to the inherent flexibility of the explicit representation, the results of the 'Hook' scene show that Grid4D has stronger ability to reconstruct fine structures than DeformGS [50] which is based on the implicit representation. We also apply the sparse control points in SC-GS [2] to our model and build SC-GS on Grid4D rather than DeformGS for further evaluation. We refer to it as 'Grid4D + SC', and observe an improvement comparison to Grid4D and SC-GS as list in the last three rows of Thanks to our 4D decomposed hash encoding, when facing the scenes with complex motions
Table I

---

Table 1: Quantitative comparisons on the synthetic D-NeRF [3 dataset. The higher PSNR (t).
higher SSIM (1) and lower LPIPS (L) denote better rendering quality. The color of each cell shows the best and the second best.
Bouncing Balls
Hell Warrior
Hook
Jumping Jacks SSIM LPIPS
Model
PSNR SSIM LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
HexPlane
40.36
0.992
0.031
24.30
0.944
0.073
28.26
0.955
0.052
31.74
0.974
0.036
Vox
40.28
0.992
0.042
27.29
0,964
0.076
30.51
0.959
0.060
33.46
0.977
0.041
4D-GS [4]
40.77
0.994
0.015
28.80
0.974
0.037
32.95
0.977
0.027
35.50
0.986
0.020
DeformGS [50]
40.91
0.995
0.009
41.34
0.987
0.024
37.06
0.986
0.016
37.66
0.989
0,013
SC-GS [12
41.59
0.995
0.009
42.19
0.989
0.019
38.79
0.990
0.011
39.34
0.992
0.008
Grid4D (Ours)
42.62
0.996
0.008
42.85
0.991
0.015
38.89
0.990
0.009
39.37
0.993
0.008
Grid4D + SC
42.17
0.995
0.008
42.81
0.990
0.017
40.26
0.992
0.008
39.58
0.993
0.008
Mutant
Standup
Trex
Mean
Model
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
PSNR
SSIM
LPIPS
HexPlane 2
33.66
0.982
0.028
34.12
0.983
0.019
31.01
0.976
0.028
31.92
0.972
0.038
ToxeuVox 8
32.07
0.961
0.048
34.46
0.980
0.033
31.43
0.967
0.047
32.78
0.972
0.050
4D-GS [
37.75
0.988
0.016
38.15
0.990
0.014
33,95
0.985
0.022
35.41
0,985
0.021
DeformGS [50]
42.47
0.995
0.005
44.14
0.995
0.007
37.56
0.993
0.010
40.16
0.991
0.012
SC-GS [12
43.43
0.996
0.005
46.72
0.997
0.004
39.53
0.994
0.009
41.65
0.993
0.009
Grid4D (Ours)
43.94
0.996
0.004
46.28
0.997
0.004
40.01
0.995
0.008
42.00
0.994
0.008
Grid4D + SC
44.07
0.996
0.004
46.87
0.997
0.004
41.12
0.995
0.008
42.41
0.994
0.008
Table Quantitative comparison on the validation rig part (Rig) and the interpolation part (Interpo- lation) of the real-world HyperNeRF [28) dataset. The higher PSNR (1) and higher MS-SSIM (t)
denote better rendering quality. The color of each cell shows the best and the second best.
Rig(4 scenes)
Interpolation66 scenes)
Model
PSNR
MS-SSIM
PSNR
MS-SSIM
TiNeu Vox g
24.20
0.836
27.08
0.922
4D-GS +
24.99
0.838
27.54
0.912
Grid4D (Ours)
25.50
0.856
28.56
0.933
TiNeuVox
4D-GS
DeformGS
GridDD (Ours)
Ground Truth
Figure 5: Qualitative comparisons on the real-world HyperNeRF
28
dataset.
and Gaussians with heavily overlapping coordinates, such as 'JumpingJacks', Grid4D predicts the deformations much more accurately than 4D-GS 44 which is built on the planed-based explicit representation relying on the unsuitable low-rank assumption.
Comparison of Rendering Speed. Comparing Frames Per Second (FPS) directly might not fair experiment because the number of Gaussians is quite different among different models. Therefore, we list both the FPS and the corresponding Gaussian count in Table 4 Despite the acceleration provided by the CUDA/C+ implementation in Grid4D, proposed explicit representation makes
8

---

Table Quantitative comparison on the real-world Neu3D [6 dataset. The higher PSNR (1) and higher SSIM (1) denote better rendering quality. The color of each cell shows the best.
Coffee Martini
Cook Spinach
Cut Beef
Flame Salmon
Flame Steak
Sear Steak
Model
PSNR
SSIM PSNR SSIM PSNR SSIM
PSNR
SSIM
PSNR
SSIM PSNR SSIM
ID-GS 444
27.34
0.898
32.50 0.942 32.26 0.942 27.99
0.902
32.54 0.951
33.44
0.954
Grid4D (Ours)
28.30
0.898
32.58
0.948
33.22
0.950
29.12
0.908
32.56
0.955
33.16
0.957
4D-GS
Gridad (Ours)
Ground Truth
Figure 6: Qualitative comparisons on the real-world Neu3D
16 dataset.
Table 4: Rendering speed comparison on the synthetic D-NeRF 31 dataset. We report the FPS
based on the number of Gaussian points. Compared to other models, our model still achieves high rendering speed and real-time rendering when facing a much larger amount Gaussians.
FPS Num(k)
Balls
Warrior
Hook
Jumping
Lego
Mutant
Standup
Trex
4D-GS
4
182 / 28
168 / 40
91 /99
207 / 24
193
173 / 38
201 / 27
151 / 68
DeformGS 50
37/ 180
161 37
43 150
71/
30/ 289
49 169
77 /
30/ 217
Grid4D (Ours)
192
334 46
79/ 210
241
64/ 302
157 126
170 100
86 254
Grid4D exhibit significantly faster rendering performance. When facing a huge number of Gaussians, Grid4D maintains high rendering speed and achieves real-time rendering. However, Grid4D has no improvement in the training speed compared DeformGS [50]. Although we do not use all Gaussians for the regularization, the smooth regularization requires Grid4D to encode the input twice, which slows down the training process. Meanwhile, we find that Grid4D's accurate deformation predictions often lead to an increase in number of Gaussians, contributing to time overhead. Nevertheless, Grid4D needs less GPU memory for training than DeformGS [50), and the extra computational cost has little influence on training in comparison to the significant improvements in rendering quality. More details can be found in Section D of our supplementury.
4.3
Ablation Study and Analysis
To mitigate the distraction of imprecise camera poses, we mainly conduct our ablation studies on the synthetic D-NeRF (31 dataset. Table 5 and Figure show the results of our ablations.
Ablation of 4D Decomposed Hash Encoding. The proposed 4D decomposed hash encoding splits the 4D input into four 3D inputs, encoding them separately without the unsuitable low-rank assumption and high space complexity. To demonstrate the advantages our encoding method, employ the simple 4D hyper-grid hash encoding in Grid4D w/o dec. The chaos in Figure 7 illustrates the rendering degradation caused by the high hash collision rate.
Ablation of Directional Attention. The directional attention helps Grid4D accurately predict the different deformations across different scene components. We scale the attention score to (0, 1) for Grid4D w/o dir to demonstrate the advantage of the directional range (-1, We also compare our attention module with the simple architecture Grid4D w/o att which directly decodes the concatenation of the spatial and temporal features. In Figure b)), the shadow has obvious different deformations

---

Table Quantitative ablation results on the synthetic D-NeRF 31 shows the best and the second best.
dataset.
The color of each cell
Model
w/o dec
w/o reg
att
w/o dir
Grid4D (Ours)
PSNR
28.45
39.47
41.37
41.32
42.00
SSIM
0,949
0,991
0.993
0.993
0.994
LPIPS
0.055
0,012
0.009
0.009
0.008
w/o dec
W'o reg (a)
Grid4D (Ours)
w/o att.
w/o dir
GridAD (Ours) Ground Truth (b)
Figure 7: Qualitative results of ablation studies. (a) is the ablation of the 4D decomposed hash encoding and the smooth regularization. The first row is the rendering results, and the second row is the visualization of the deformation. The similar colors in the deformation map mean similar deformation sizes on each axis. (b) is the ablation the directional attention.
from the neighboring parts across the timeline. Our directional attention achieves high clarity in rendering the portion with the shadow, emphasizing its effectiveness capturing such variations.
Ablation of Smooth Regularization. The proposed smooth regularization aims at mitigating the chaos of deformation prediction. We train Grid4D without the smooth regularization in Equation and refer to the model as Grid4D w/o reg. The results in Figure aa) show that the regularization reduces the deformation artifacts caused by the lack of smoothness.
We conduct more ablation studies for the architecture and smooth regularization. We also visualize the intermediate results of our model. More results can be found in Section Cf supplementary.
5 Conclusion
In this paper, we have introduced Grid4D, novel model for high-fidelity dynamic scene rendering. Grid4D utilizes the proposed 4D decomposed hash encoding without the unsuitable low-rank as- sumption and high space complexity. Additionally, the novel directional attention module effectively aggregates the spatial and temporal features for more accurate deformation prediction across different scene components. Moreover, employed smooth regularization to mitigate chaos in deformation prediction, resulting in high rendering quality. Our experiments demonstrate that Grid4D achieves state-of-H the-art performance and delivers high rendering speed for dynamic scene rendering. However, Grid4D has no improvement in training speed, and like the other dynamic scene rendering models, Grid4D might have artifacts when facing several dynamic scenes with complex and large motions. Addressing these challenges remains an area for future research.
Acknowledgements
The authors would like to thank the anonymous reviewers their critical and constructive comments and suggestions. The authors also thank The Supercomputing Center of Nankai University (NKSC) for computation. This work was supported by the National Science Fund of China under Grant No. 66661166670 and Grant No. 62276144.
10

---

References
[1] Hongrui Cai, Wanquan Feng, Xuetao Feng. Yan Wang, and Juyong Zhang. Neural surface reconstruction of dynamic scenes with monocular RGB-D camera. Advances Neural Information Prucessing Systems. 35:967-981, 2022.
[2] Ang Cao and Justin Johnson. HexPlane: A fast representation for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 130-141, 2023.
[3] Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3D generative adversarial networks. In arXiv, 2021.
[4] Jaesung Choe, Christopher Choy, Jaesik Park, In So Kweon, and Anima Anandkumar. Spacetime surface regularization for neural dynamic scene reconstruction. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 17871-17881, 2023.
[5] Yilun Du, Yinan Zhang, Hong-Xing Yu, Joshua Tenenbaum, and Jiajun Wu. Neural radiance flow for 4D view synthesis and video processing. In 202/ IEEE/CVF International Conference on Computer Vision (ICCV), pages 14304-14314. IEEE Computer Society, 2021.
[6] Yuanxing Duan, Fangyin Wei, Qiyu Dai. Yuhang He, Wenzheng Chen, and Baoquan Chen. 4D Gaussian Splatting: Towards efficient novel view synthesis for dynamic scenes. arXiv preprint arXiv:2402.03307. 2024.
[7] Bardienus Duisterhof, Zhao Mandi, Yunchao Yao, Wei Liu, Mike Zheng Shou, Shuran Song, and Jeffrey Ichnowski. MD-Splatting: Learning metric deformation from 4D Gaussians in highly deformable scenes. arXiv preprint 2rX1v:2312.00583, 2023.
[8] Jiemin Fang, Taoran Yi, Xinggang Wang, Lingxi Xie, Xiaopeng Zhang, Wenyu Liu, Matthias Nießner, and Qi Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, pages 9, 2022.
[9] Sara Kridovich-Keil, Giacomo Meanti, Frederik Rahbæk Warburg, Benjamin Recht, and Angjoo Kanazawa. K-Planes: Explicit radiance fields in space, time, and appearance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12479-12488, 2023.
[10] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang, Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712 2221, 2021.
[11] Xiang Guo, Jiadai Sun, Yuchao Dai, Guanying Chen, Xiaoqing Ye, Xiao Tan, Errui Ding, Yumeng Zhang, and lingdong Wang. Forward flow for novel view synthesis of dynamic scenes. In Proceedings of the IEFF/CVF International Conference Computer Vision, pages 16022-16033, 2023.
[12] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang. Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. SC-GS: Sparse-controlled Gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.1493: 2023.
[13] Bernhard Kerbl, Georgios Kopanas. Thomas Leimkühler, and George Drettakis. 3D Gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics, 42(4), 2023.
[14] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980) 2014.
[15] Agelos Kratimenos, Jiahui Lei, and Kostas Daniilidis. DynMF: Neural motion factorization for real-time dynamic view synthesis with 3D Gaussian splatting, arXiv preprint arXiv:2312.00112 2023.
[16] Tianye Li, Mira Slaveheva, Michael Zollhoefer, Simon Green, Christoph Lassner, Changil Kim. Tanner Schmidt, Steven Lovegrove, Michael Goesele, Richard Newcombe, et al. Neural 3D video synthesis from milt video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[17] Zhan Li, Zhang Chen, Zhong Li. and Yi Xu. Spacetime Gaussian feature splatting for real-time dynamic view synthesis. arXiv preprini arXiv:2312.168/2 2023.
[18] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 6498-6508, 2021.

---

[19] Zhengqi Li, Qiangian Wang, Forrester Cole, Richard Tucker, and Noah Snavely. DynlBaR: Neural dynamic image-based rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 4273-4284, 2023.
[20] Yiqing Liang, Numair Khan, Zhenggin Li. Thu Nguyen-Phuoc, Douglas Lanman, James Tompkin, and Lei Xiao. GauFRe: Gaussian deformation fields for real-time dynamic novel view synthesis. arXiv preprint arXiv:2312.1 1158,, 2023.
[21] Yiqing Liang, Eliot Laidlaw, Alexander Meyerowitz, Srinath Sridhar, and James Tompkin. Semantic attention flow fields for monocular dynamic scene decomposition. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 21797-21806, 2023.
[22] Youtian Lin, Zuozhuo Dai, Siyu Zhu, and Yao Yao. Gaussian-Flowe 4D reconstruction with dynamic 3D Gaussian particle. arXiv preprint arXiv:2312.0344 2023.
[23] Yu-Lun Liu, Chen Gao, Andreas Meuleman, Hung- Yu Tseng, Ayush Saraf. Changil Kim. Yung- Yu Chuang. Johannes KopI. and Jia-Bin Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13-23, 2023.
[24] Zhicheng Lu, Xiang Guo, Le Hui, Tianrui Chen, Min Yang, Xiao Tang. Feng Zhu, and Yuchao Dai. 3D geometry-aware deformable Gaussian splatting for dynamic view synthesis. arXiv preprint mmXiv:2404.062700 2024.
[25] Jonathon Luiten, Georgios Kopanas, Bastian Leibe, and Deva Ramanan. Dynamic 3D Gaussians: Tracking by persistent dynamic view synthesis. arXiv preprint arXiv:2308.0977 2023.
[26] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. NeRF: Representing scenes neural radiance fields for view synthesis. Communications of the ACM, 65(1))99-110) 106, 2021.
[27] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller. Instant neural graphics primitives with multiresolution hash encoding, ACM Transactions Graphics (ToG), 41 2022.
[28] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz. HyperNeRF: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2 13228, 2021.
[29] Sungheon Park, Minjung Son, Seokhwan Jang. Young Chun Ahn. Yeon Kim, and Nahyup Kang. Temporal interpolation is all you need for dynamic neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 4212-42211, 2023.
[30] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. PyTorh: An imperative style. perforrerfmmeece deep learning library. Advances in neural information processing systems, 32, 2019.
[31] Albert Pumarola, Enric Corona, Gerard Pons- Moll, and Francesc Noguer- D-NeRF: Neural radiance fields for dynamic scenes. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318-10327, 2021.
[32] Johannes L Schonberger and Jan-Michael Frahm. Meneture-froo-Mooiiee revisited. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-4113, 2016.
[33] Ruizhi Shao, Zerong Zheng. Hanzhang Tu, Boning Liu, Hongwen Zhang, and Yebin Liu. Tensor4D: Efficient neural 4D decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16632-16642, 2023.
[34] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409. 1556, 2014.
[35] Nagabhushan Somraj, Kapil Choudhary, Sai Harsha Mupparaju, and Rajiv Soundararajan. Factorized motion fields for fast sparse input dynamic view synthesis. arXiv preprint arXiv:2404. 1669, 2024.
[36] Liangehen Song, Anpei Chen, Zhong Li, Zhang Chen, Lele Chen, Junsong Yuan, Yi Xu, and Andreas Geiger. NeRFPlayer: A streamable dynamic scene representation with decomposed neural radiance fields. IEEE Transactions on Visualization and Computer Graphics. 29(5):233--2753 2023.
[37] Jiakai Sun, Han Jiao, Guangyuan Li, Zhanjie Zhang, Lei Zhao, and Wei Xing. 3DGStream: On-the-fly training of 3D Gaussians for efficient streaming of photo-realistic free-viewpon viewpoint videos. arXiv preprint arXiv:2403.0144 2024.
12

---

[38] Edgar Tretschk, Ayush Tewari. Vladislav Golyanik, Michael Zollhöfer, Christoph Lassner, and Christian Theobalt. Non-Rigid Neural Radiance Fields: Reconstruction and novel view synthesis of dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19959- 2021.
[39] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-SNE. Journal of machine learning research, 9(11), 2008.
[40] Chaoyang Wang, Lachlan Ewen MacDonald, Laszlo A Jeni, and Simon Lucey. Flow supervision for de- formable NeRF. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21 128 21137, 2023.
[41] Feng Wang, Zilong Chen, Guokang Wang. Yafei Song, and Huaping Liu. Masked space-time hash encoding for efficient dynamic seene reconstruction. Advances in Neural Information Processing Systems, 2023.
[42] Feng Wang. Sinan Tan, Xinghang Li, Zeyue Tian, Yafei Song. and Huaping Liu. Mixed neural voxels for fast mueti-view video synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 19706-19715 2023.
[43] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4);600 612, 2004.
[44] Guanjun Wu, Taoran Yi. Jiemin Fang. Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian. and Xinggang Wang. 4D Gaussian splatting for real-time dynamic scene rendering, arXiv preprint arXiv:2310.08526 2023.
[45] Ziyang Xie, Junge Zhang, Wenye Li. Feihu Zhang, and Li Zhang. S-NeRF: Neural radiance fields for street views. arXiv preprint arXiv:2303.00749 2023.
[46] Zhen Xu, Sida Peng, Haotong Lin, Guangzhao He, Jiaming Sun, Yujun Shen, Hujun Bao, and Xiaowei Zhou, 4K4D: Real-time 4D view synthesis at 4K resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20029 20120, 2024.
[47] Zhiwen Yan, Chen Li, and Gim Hee Lee. NeRF-DS: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9285-8295. 2023.
[48] Gengshan Yang. Minh Vo, Natalia Neverova, Deva Ramanan, Andrea Vedaldi, and Hanbyul Joo. BANMo: Building animatable 3D neural models from many casual videos. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 2863-2873, 2022.
[49] Zeyu Yang, Hongye Yang. Zijie Pan, Xiatian Zhu, and Li Zhang. Real-time photorealistic dynamic scene representation and rendering with 4D Gaussian splatting, arXiv preprint arXiv:2310. 10642, 2023.
[50] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3D Gaussians for high-fidelity monocular dynamic seene reconstruction. arXiv preprint arXiv:2309.13101 2023.
[51] Wang Yifan, Felice Serena, Shihao Wu, Cengiz Öztireli, and Olga Sorkine-Horunn, Differentiable surface splatting for point-hased geometry processing. ACM Transactions on Graphics (TOG), 38(6):1-14, 2019.
[52] Heng Yu, Joel Julin, Zoltán Á Milaeski, Koichiro Niiumma, and László A Jeni. CoGS: Controllable Gaussian splatting. arXiv preprini arXiv:2312.05666 2023.
[53] Wentao Yuan, Zhaoyang Lv, Tanner Schmidt, and Steven Lovegrove. STaR: Self-super tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23144-3112. 2021.
[54] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586 595. 2018.
[55] Ruiqi Zhang and Jie Chen. NDF: Neural deformable fields for dynamic human modelling. In European Conference on Computer Vision, pages 37-52. Springer, 2022.
[56] Matthias Zwicker, Hanspeter Plister, Jeroen Van Baar, and Markus Gross. Surface splatting. In Proceedings of the 28th annual conference Computer graphics and interactive techniques, pages 371-378, 2001.
13

---

Supplementary
A Details of Experimental Setup
Network Architecture of Grid4D. The upper limit of the hash table size is 2 for both spatial and temporal 3D grids. The feature dimension each voxel is for all grids. We set the spatial dimension resolutions of temporal grids according to the scale of the scene. We usually set the time dimension resolution to a value between half and quarter of the time samples. The architecture of our multi-head directional attention decoder is illustrated by Figure g The spatial and temporal MLPs only have one fully connected layer and one activation layer. For the multi-head deformation decoder, we set the depth to two for the HyperNeRF [28] dataset and one for the D-NeRF [3]] dataset, including the output layer, and set all the widths to 256.
Spucill Peeture
Leas
Terpprri Frutarn
Product
As
Figure 8: Architecture of our multi-head directional attention decoder.
Optimization. The scheduler of the learning rate primarily follows DeformGS [50] 13
The loss
weight de and X, in Equation is set to 0.2 and 0.5 for common scenes. Notably, the learning rate of the MLP decoder is determined based on the scale of the scene. Additionally, the learning rate of the grid hash encoders is set to 10-50 times larger than the MI.P decoder. We use Adam [14] optimizer with 3 = (0.9, 0.999) for training and set the background to black. Due to the differences between the spatial and temporal grids,
set different smooth regularization parameters for the (x, y, z) grid in several scenes. For the scenes in the HyperNeRF (28) and Neu3D [16] dataset, we
use the SiM [32] points to initialize Gaussians.
Deformation Map. The formula for deformation maps
Ar f(x, -
(10)
T
where v is the canonical Gaussian position, and f the deformation field. For all experiments. We set to 0.05, and limit the absolute value of Az for color. This map shows the situation of predicted deformation: the similar colors two parts mean similar deformation sizes on each axis.
B Additional Comparisons
Additional Results on D-NeRF (31] Dataset. We visualize more experimental results the D-
NeRF [3I] dataset in Figure 13 We observe that Grid4D exhibits superior rendering quality compared the state-of-the-arr models.
Per Scene Results on HyperNeRF 28 Dataset. We provide the per-scene results for the experiments the real-world HyperNeRF 28 dataset. Table an Figure illustrate the comparisons. While the quantitative results for Grid4D do not surpass those of other models in several scenes, is noteworthy that our model exhibits significantly improved clarity in rendering, demonstrated in Figure an Figure 12 We also find reconstruction failures of DeformGS [50] in the 'Teapot' and 'Broom" scenes (the second and last line of Figure TOp, as mentioned in their paper.
C Additional Ablations
Additional Architecture Ablations. We also conduct more ablation studies for Grid4D. We change the depth of the multi-head decoder to one and two for the D-NeRF dataset, and the max level apply the simple position
number I. of the temporal grid hash encoder to 8 and 16. Additionally, deformation method to Grid4D as Grid4D w/o RT, which directly adds the deformation the position
and is used in the prior works 44 50). The results can be found in the Table 7 We can find that
14

---

Table Per scene comparisons on the real world HyperNeRF 22 'vrig' and 'interp' dataset. The higher PSNR (1) and higher MS SSIM t denote better rendering quality. The color of each cell
shows the best and the second best. We set all rendering resolutions 536 900.
3D Printer
Broom
Chicken (vrig)
Teapot PSNR MS-SSIM PSNR MS-SSIM
Poel Banana
Model
PSNR MS-SSIM PSNR MS-SSIM PSNR MS-SSIM
Tox g
22.77
0.839
21.27
0.683
28.27
0.948
24.50
0.874
24.15
0.893
4D-G3 DD-GS [4
22.00
0.807
21.80
0.684
28.55
0.927
27.62
0.934
26.99
0.941
Grid4D (Ours)
22.35
0.825
21.86
0.710
29.26
0.942
28.53
0.946
26.53
0.932
Chicken (interp)
Cut I Lemon
Hand
Slice Banana
Chocolate
Model
PSNR MS-SSIM
PSNR MS-SSIM PSNR MS-SSIM
PSNR MS-SSIM
PSNR MS-SSIM
Tox 8
27.69
0,951
28.54
0.955
27.44
0.872
27.67
0.916
26.97
0.948
4D-GS +
26.91
0.912
30.26
0.936
29.87
0.939
25.18
0.812
26.05
0.933
Grid4D (Ours)
27.31
0.924
32.18
0.967
31.31
0.958
25.79
0.853
28.23
0.965
Table 7: Additional ablation results of model architecture on the synthetic D-NeRF 31 dataset. The color of each cell shows the best and the second best.
Model L = 8, d 0 L 16, d = L 32, d = 0 (Ours) == 32, d =
L = 32, d = 2 w/o RT
PSNR (t)
41.34
41.69
42.00
41.96
41.75
41.80
Table 8: Comparison of average training computational on the D-NeRF 31
dataset with single
RTX 3090 GPU.
Model
4D-GS 44
DeformGS 50
SC-GS 12
Grid4D (Ours)
Time
20min
33min
75min
55min
GPU Memory
1GB
4.5GB
3.IGB
4.0GB
PSNR
35.41
40.16
41.65
42.00
when the model becomes deeper, the performance might become worse, and we consider that the reason might be the training difficulties of deep MLPs.
Additional Smooth Regularization Ablations. We conduct a smooth regularization for the both grid hash encoder and MLP decoder in Grid4D w both by adding the following loss,
La Rg R2+el12
112 + T T2-
(11)
where RG Tr are the predicted position deformation of the 4D input and Ry+e, TeseeTz- Tz-e are the deformation of the perturbed 4D input x + e. The results are shown in the part Table and we can find that the smooth regularization of the MLP decoder does not make sense. We consider that this is because of the smooth inherent property of MLPs.
Visualization of Feature, Deformation, and Depth Maps. We also visualize the feature maps by pro- jecting the L2 norm of the features encoding the Gaussian positions from our 4D decomposed hash en- coder. We set the RGB color of the temporal feature map to the I.2 norm of (a,, y, t), (y. 2, tt, (x, t) grid features, and the feature map is rendered by the rasterization of Gaussians with the specified color. The results in Figure [T denote that the proposed encoding method effectively represents the deformation features in both temporal and spatial spaces. Also, the depth maps show that we have a precise depth prediction.
D Limitation
Although our model achieves tfate-of-the-art performance with our proposed explicit representation, Grid4D has improvement in training speed. However, compared to DeformGS (50), Grid4D has less memory overhead. As shown in Table the computational cost has little influence on model training. Figure 9 displays several artifacts when Grid4D and the tfte-of-the-ar models 4 50 render several scenes with large and complex motions.
15

---

Table 9: Additional ablation results of adding MLP decoder regularization on the D-NeRF dataset.
Model
Grid4D w both
Grid4D (Ours)
PSNR (t)
41.92
42.00
4D-GS
DeformGS
Grid4D (Ours) Ground Truth
4D-GS
DefomGS
Grid4D (Ours) Ground Truth
Figure 9: Artifacts of Grid4D and other state-of- sute-of-theeart models
50
TiNeuVox
4D-GS
DeformGS
Grid4D (Ours)
Ground Truth
Figure 10: Additional qualitative comparisons on the real-world HyperNeRF
28
dataset.
16

---

Figure 11: Additional analysis of Grid4D. Each line in turn shows the temporal feature maps, spatial feature maps, deformation maps, and depth maps. The lighter parts in the feature maps denote the stronger activation of the corresponding features.
TiNeuVox
4D-GS
DeformGS
Grid4D (Ours)
Ground Truth
Figure 12: Additional qualitative comparisons on the real-world HyperNeRF
28 dataset.
17

---

scc00
Bouncing Balls
Hell Wamor
Hook
Jumping Jacks
Lego
Mulantt
PERER
Slanddup
Trex
TiNeuV ox
4D-GS
DeformGS
Grid4D (Ours)
Ground Truth
Figure 13: Additional qualitative comparisons on the synthetic D-NeRF 31
dataset.
18

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We accurately claim the contributions and scope of this paper in the abstract and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We discuss the insignificant limitation of training efficiency and complex motion fitting in Section and Section of the supplementary.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
19

---

Answer:
[NA]
Justification: The proposed methods are mainly experimental, and we demonstrate the effectiveness of our methods through abundant experiments.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We describe our methods clearly in Section 3 and Section 4.1 More details can be found in the supplementary, and we will release our codes and checkpoints if the paper is accepted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
20

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We will release all codes the GitHub platform, and provide the URL in the camera-ready version. We will offer the instructions to reproduce the results in detail.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
http:/.cccpubllc/uuuuccee Codeeubmii ssi criionPooiccy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provide important settings and an overview of the experimental setup released.
in Section 4.1 More details can be found in Section of the supplementary code
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The experiments do not have obvious experimental statistical significance. However. we directly provide the detailed experimental results every scenes.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
21

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We provide the GPU used in our experiments. We also describe the detailed compute resources of methods in Section D Df the supplementary and Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics htttge///oorrpp..ccpuuu///ttiii
Answer: [Yes]
Justification:
The research is strictly conducted with the NeurIPS Code of Ethics in every
respect.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work does not have obvious societal impacts.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
22

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We use creditable assets and properly respect them in Section 4.1
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
23

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, batasets.papersstt paperswi thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We describe the used public datasets in Section 4.1 and will release code for academic usage.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The research is not with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The research is not with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
25


Reviews:
[{"review_id": "2ZdhBc5OAd", "reviewer": "Reviewer_2Pr7", "summary": "The paper introduces Grid4D, a novel dynamic scene rendering model that leverages hash encoding for 4D input decomposition, enabling high-quality and speedy rendering of dynamic scenes. \nUnlike traditional plane-based methods that suffer from excessive feature overlap due to low-rank assumptions, Grid4D uses a tri-axial 4D decomposition into spatial and temporal 3D hash encodings. \nThis approach, combined with a novel directional attention mechanism, improves the discrimination and aggregation of deformation features, enhancing both the visual quality and rendering speed of dynamic scenes.", "questions": "1. How does Grid4D handle scenes with highly non-uniform motion patterns?\n\n2. Could the directional attention mechanism be adapted for use in other types of neural network architectures beyond scene rendering?\n\n3. What are the specific challenges faced when applying Grid4D to real-world datasets compared to synthetic ones?\n\n4. How does the smooth regularization impact the model's performance in terms of real-time rendering capabilities?", "limitations": "1. Grid4D shows limited improvement in training speed.\n\n2. It does not fully address challenges related to extremely dynamic or unpredictable environments.\n\n3. The effectiveness of the model heavily relies on the novel encoding and attention mechanisms, which might not translate as effectively to different rendering tasks or simpler scene compositions.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Grid4D's use of 4D decomposed hash encoding addresses the limitations of plane-based methods by effectively reducing feature overlap, which enhances feature discriminability and rendering accuracy. The novel attention mechanism aligns well with the diverse deformations across different scene components, allowing for more accurate deformation prediction and rendering. It is also good to demonstrate superiority over SoTA models.", "weaknesses": "1. Despite its advancements in rendering performance, Grid4D does not significantly improve the training speed compared to existing models.\n\n2. The sophisticated architecture involving multiple hash encodings and the directional attention mechanism might complicate the implementation and tuning of the model.\n\n3. The model's complexity and the specific tuning required for different datasets might limit its generalizability or lead to overfitting on particular types of dynamic scenes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. We hope that our response below will address your concerns.\n\n**Q1: Grid4D does not significantly improve the training speed in comparison to existing models.**\n\nA1: Although our model has no improvement in training speed, our model can obtain a good balance between time consumption and performance. Additionally, the GPU memory consumed with our model is comparable to DeformGS and SC-GS. We report the average time, GPU memory and PSNR on the D-NeRF dataset in the following table.\n\n|    Model     | Training Time | GPU Memory | FPS  | PSNR  |\n| :----------: | :-----------: | :--------: | :--: | :---: |\n|    4D-GS     |     20min     |    1GB     | 160  | 34.11 |\n|   DeformGS   |     33min     |   4.5GB    |  62  | 38.26 |\n|    SC-GS     |     75min     |   3.1GB    | 179  | 39.56 |\n| Grid4D(Ours) |     55min     |   4.0GB    | 153  | 39.86 |\n\n**Q2: Will the complexity and specific tuning of Grid4D limit its generalizability ?**\n\nA2: Although our model has multiple hash encodings, it has fewer reconstruction failures than DeformGS on the real-world HyperNeRF dataset, as shown in 'Teapot' (last row of Figure 9 in our paper) and 'Hand' scenes (middle row of Figure 11 in our paper).\n\nTo demonstrate the versatility of our model, we also apply Grid4D to the Neu3D[1] dataset which has larger scenes and more complex motions. We report PSNR in the following table. The qualitative results can be found in Figure 1 of the top comment PDF. From this table, one can see that although our model has multiple hash encodings, our model can still yield better performance. We will add these experiments to our paper in the final version.\n\n|    Model     | Coffee Martini | Cook Spinach | Cut Beef  | Flame Salmon | Flame Steak | Sear Steak |   Mean    |\n| :----------: | :------------: | :----------: | :-------: | :----------: | :---------: | :--------: | :-------: |\n|    4D-GS     |     27.34      |    32.50     |   32.26   |    27.99     |    32.54    | **33.44**  |   31.01   |\n| Grid4D(Ours) |   **28.30**    |  **32.58**   | **33.22** |  **29.12**   |  **32.56**  |   33.16    | **31.49** |\n\nThe main tuning of our model is the resolution of the temporal hash encoder because different scenes have different numbers of frames. Additionally, according to the Nyquist-Shannon Sampling Theorem, the temporal resolution can be set to half or a quarter of the frame number automatically by the algorithm.\n\n**Q3: How does Grid4D handle scenes with highly non-uniform motions ?**\n\nA3: Firstly, explicit representation methods are much more flexible than implicit ones, making Grid4D better suited for modeling highly non-uniform motions. Secondly, our hash encoder decomposes 4D coordinates into four 3D coordinates. When encoding heavily overlapping 4D inputs, the 3D decomposition results will have less same parts. Therefore, the encoded explicit features will have fewer overlapped parts, resulting in a more accurate representation of the corresponding non-uniform motions.\n\nIn our experiments, the 'Hand', 'Peel Banana', and 'Chocolate' scenes in HyperNeRF dataset have much more highly non-uniform motions. As listed in Table 5, Figure 9, and Figure 10 of our paper, our methods can significantly achieve better rendering quality than the state-of-the-art models.\n\n**Q4: Can the directional attention mechanism be adapted for other usage ?**\n\nA4: We consider that directional attention could also be used in 4D generation models for temporal feature selection. For example, we could generate the directional attention score from the features unrelated to the timestamp, and apply the score to the other features.\n\n**Q5: What are the specific challenges faced when applying Grid4D to real-world datasets in comparison to synthetic ones?**\n\nA5: The main challenge is that the real-world scenes usually have imprecise camera poses which might lead to the failure or degradation of rendering. Another challenge is that real-world scenes usually have more complex motions.\n\nFor the imprecise camera poses, the smooth regularization term helps model keep consistency in the neighboring region. Such smoothness makes the model tend to fit the average results captured by the imprecise camera poses, which improves the robustness of our model. We can increase the weights of the smooth regularization term to handle more imprecise camera poses in the real-world datasets.\n\nTo handle complex motions, our model improves the discriminability of the explicit features. Encoded by our 4D decomposed hash encoder, our explicit features have fewer overlapped parts, resulting in a more accurate representation of the corresponding non-uniform motions. We can increase the resolution of 4D decomposed hash encoder to model more complex motions in the real-world datasets.\n\n**Q6: How does the smooth regularization impact the real-time rendering capabilities of the model ?**\n\nA6: The smooth regularization does not affect real-time rendering capabilities. During the rendering period, the smooth regularization term and other losses are not calculated by the model. Smooth regularization is mainly used in the training process and improve the rendering quality.\n\n[1] Li, Tianye, et al. \"Neural 3d video synthesis from multi-view video.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IUYwQsZIjs", "reviewer": "Reviewer_V3gV", "summary": "The paper presents a grid-based method to compute deformed gaussians to render dynamic scenes from input images. It proposes to perform a 3D decomposition of the 4D input, using multiresolution hash encoding to access spatial (static) and temporal (dynamic) feature vectors. The static features are fed to an MLP responsible of computing a directional attention score, which is composed with temporal features coming from another MLP, resulting in the deformation features. Those features pass through a multi-head decoder, which infer the deformation parameters. Those parameters are used to deform the set of canonical gaussians, which are rasterized to generate the final image.", "questions": "(1) Small presentation advice: at the introduction, it would be better to talk about Figure 1 before Figure 2, because Figure 1 is very good as a primary description of how the 4D encoding is decomposed. Starting with Figure 2 is a little bit more confusing.\n\n(2) I think the Preliminaries section should include a small introduction to multiresolution hash encoding too. It would help the reader to have better context for the remaining of the method section.\n\n(3) I would like to know the values of $\\lambda_c$ and $\\lambda_r$.", "limitations": "Yes, the limitations are presented in the supplementary.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "(1) Contextualization is very good, cited works are sufficient. The Introduction and Related Work sections may be successfully used for familiarization with the fields needed for proper understanding of the paper.\n\n(2) Every part of the method is intuitively presented and the Math holds. \n\n(3) Mathematical notation is very clean.\n\n(4) I liked the introduction of an attention mechanism in this context and the idea behind the smooth regularization.", "weaknesses": "(1) I am not convinced about the temporal smoothness and coherence of the proposed method. The interpolations needed for grid-based methods are known to introduce discontinuities even in static cases (NGLOD, Instant-NGP). The paper does not present a supplementary video nor a temporal coherence metric (such as FID-VID or FVD). Moreover, all presented experiments are based on specific frames of the datasets. It is not currently possible to qualitatively or quantitatively evaluate this aspect of the proposed approach, which is crucial for a 4D method. This is the core reason behind my current rating.\n\nSince the authors are not allowed to add a supplementary video nor include a link in the rebuttal (as far as I know), I believe a way to answer this critic with data would be to show a table with temporal coherence metrics. However, I encourage the authors to present other arguments if they find creative ways to be convincing. I liked the paper and I am willing to increase my rating in case the authors are convincing in this subject.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. We hope that our response below will address your concerns.\n\n**Q1: How to demonstrate the temporal coherence of the rendering results ?**\n\nA1: To demonstrate the temporal coherence of our model, we provide some videos by displaying included images in Figure 3 of the top comment PDF. We uniformly interpolate the time and randomly select a camera pose to render the scene reconstructed by Grid4D, and select the images per 5 frames (full 150 frames) to display. We will provide more videos on our GitHub repository in the final version.\n\nIn the test stage of the dynamic scene rendering task, the camera poses and timestamps are usually not selected in a video sequence. To our knowledge, there has no existing metric to evaluate the temporal coherence of a video without the ground truth. To further demonstrate the temporal coherence of the rendering results, we attempt to design a simple metric to evaluate the coherence. We use the optical flow estimation model RAFT[1] to predict the forward and backward flow between the two neighbouring frames of the rendered video. Similar to Chamfer Distance, we calculate the mean of total pixel displacements, as the following formula,\n\n$F^{forward} = \\frac{1}{NHW}\\sum_{i, h, w}||f_{ihw}^{forward}||_2$\n\n$F^{backward} = \\frac{1}{NHW}\\sum_{i, h, w}||f_{ihw}^{backward}||_2$\n\n$F = \\frac{1}{2}(F^{forward}+F^{backward})$\n\nwhere $N, H, W$ denote to the frames, height, width of the video, $f$ means the corresponding flow. The unit of $F$ is pixel. The lower value of $F$ denotes the better temporal coherence keep in the video.\n\nWe use the official pretrained RAFT[1] model of PyTorch to estimate the flow. We uniformly interpolate 150 timestamps along the timeline and randomly select a camera pose for our model to render the video. The results on the D-NeRF dataset are listed in the following table. For reference, we random select a video from the Kinetics-400[2] dataset and calculate the same metric $F$.\n\n| Bouncing Balls | Hell Warrior | Hook | Jumping Jacks | Lego  | Mutant | Standup | Trex  | *Reference* |\n| :------------: | :----------: | :--: | :-----------: | :---: | :----: | :-----: | :---: | :--: | \n|     0.248      |    0.673     | 1.014 |     1.799     | 0.220 | 0.324  |  1.331   | 0.576 | *2.137* |\n\nAccording to the results, the average pixel displacements of the rendered videos are all below 2.0pix and could demonstrate the temporal coherence of our rendering results. However, the metrics mentioned above are not well explored which might need further research.\n\n**Q2: What are the values of $\\lambda_c$ and $\\lambda_r$ ?**\n\nA2: In our experiments, $\\lambda_c$ and $\\lambda_r$ are set as follows: $\\lambda_c=0.2$ for all scenes. $\\lambda_r=0.5$ for most scenes, $\\lambda_r=1.0$ for the 'vrig' part of HyperNeRF dataset.\n\nIn most cases, set $\\lambda_r=0.5$ can obtain reliable results. For better performance, increasing $\\lambda_r$ can help Grid4D render better when the scene primarily consists of rigid objects and simple motions. Similarly, decreasing $\\lambda_r$ can help Grid4D for modeling complex motions and non-rigid objects.\n\n**Q3: Problems of writing.**\n\nA3: We will add the introduction of Multi-resolution Hash Encoding and the description of Figure 1 in the final version.\n\n[1] Zachary Teed and Jia Deng. Raft: Recurrent all-pairs field transforms for optical flow. In Computer Vision–ECCV 2020:16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, pages 402–419. Springer, 2020.\n\n[2] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950,2017.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SskM11lBda", "reviewer": "Reviewer_KYXG", "summary": "This paper proposes a  Grid4D representation for dynamic scene rendering. It breaks low-rank assumptions on 4D-GS and propose a decomposed 4D hash-grid representation for encoding canonical 3D Gaussians. A attention module is used to aggregate the spatial-temporal features of 3D Gaussians. More training strategies are applied in Grid4D to ensure efficient training and rendering quality are also better than 4D-GS, and deformable-GS.", "questions": "1. I want the authors to demonstrate that Grid4D is better than 4D-GS, Deformable-GS, and SC-GS in most evaluation metrics. Please use average results instead of the range 0.3~1.5 in Table 7, as that range is too large. If some metrics are worse, please provide more discussions.\n2. Solving a low-rank problem to achieve better rendering quality in the D-NeRF/NeRF-DS/HyperNeRF dataset is indeed a contribution. However, the reviewer believes that the core challenges of novel view synthesis in this dataset have already been solved. There are many important topics in dynamic scene novel view rendering, such as large motion and monocular novel view synthesis.", "limitations": "As mentioned in weakness and questions.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a deformable representation for dynamic scene rendering and without relying on low-rank assumption.\n2. The attention module looks reasonable for feature aggregation.\n3. A smooth training strategy is employed to ensure the smoothness of Grid4D.", "weaknesses": "1. Why don’t you compare with [13]? SC-GS now preserves the state-of-the-art (SOTA) rendering quality in D-NeRF.\n2. If you choose 4D-GS as your baseline, please provide the complete results on the Neu3D dataset and all other metrics (average training time, average rendering speed, average storage consumption).\n3. Only selecting some figures and reporting PSNR/LPIPS/SSIM for comparison is not quite enough in 3D vision. The reviewer recommends that the authors submit more comparison videos as supplementary material.\n4. If the authors want to propose a method that can replace the deform-based dynamic Gaussians (4D-GS, SC-GS, deform-GS) and become a new baseline, they should provide strong, comprehensive results in the experiment section.\n5. Additionally, the D-NeRF dataset is just a toy. I don't think much efforts are deserved to used in the dataset after the publication of SC-GS (PSNR already up to 40). If the authors are still interested in dynamic scene novel view rendering, I recommend focusing on monocular setups (DyCheck, NeurlPS2022) or multi-view (Neu3D, CVPR 2022).\n6. The figure looks unattractive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. We hope that our response below will address your concerns.\n\n**Q1: Comparison to SC-GS.**\n\nA1: We conduct the comparison with SC-GS on the D-NeRF dataset. The comparison results with PSNR can be found in the following table, and the qualitative results can be found in Figure 2 of the top comment PDF. Notably, the original SC-GS rendering resolution is $400\\times 400$, lower than our model, so we changed the rendering resolution of SC-GS to $800\\times 800$ for fairness. Additionally, we remove the 'Lego' scene because of the incorrect test ground truth. We will add these experiments in the final version.\n\n|    Model     | Bouncing Balls | Hell Warrior |   Hook    | Jumping Jacks |  Mutant   |  Standup  |   Trex    | Mean      |\n| :----------: | :------------: | :----------: | :-------: | :-----------: | :-------: | :-------: | :-------: | --------- |\n|    SC-GS     |     41.59      |    42.19     |   38.79   |     39.34     |   43.43   | **46.72** |   39.53   | 41.65     |\n| Grid4D(Ours) |   **42.62**    |  **42.85**   | **38.89** |   **39.37**   | **43.94** |   46.28   | **40.01** | **42.00** |\n\nAccording to the results, Grid4D has better performance than SC-GS on average. The improvement of Grid4D might be because of the performance gap between implicit deformation fields (which SC-GS is based on) and Grid4D. SC-GS uses a full MLP-based implicit model to predict deformations, which has the over-smooth inherent property. Our proposed 4D decomposed hash encoder generates explicit features with high discriminability to represent the deformations more accurately.\n\nWe think that SC-GS is an excellent work mainly focused on dynamic scene editing and rendering. Therefore, combining the advantages of SC-GS and Grid4D or applying Grid4D to SC-GS are probably great future works that might get better rendering quality.\n\n**Q2: Results on Neu3D dataset.**\n\nA2: We conduct the experiments on the Neu3D dataset and report PSNR in the following table. From this table, one can see that our model has better performance than the state-of-the-art model. The qualitative results can be found in Figure 1 of the top comment PDF. We will add these experiments to our paper in the final version.\n\n|    Model     | Coffee Martini | Cook Spinach | Cut Beef  | Flame Salmon | Flame Steak | Sear Steak |   Mean    |\n| :----------: | :------------: | :----------: | :-------: | :----------: | :---------: | :--------: | :-------: |\n|    4D-GS     |     27.34      |    32.50     |   32.26   |    27.99     |    32.54    | **33.44**  |   31.01   |\n| Grid4D(Ours) |   **28.30**    |  **32.58**   | **33.22** |  **29.12**   |  **32.56**  |   33.16    | **31.49** |\n\n**Q3: Can the authors provide comparison videos ?**\n\nA3: To demonstrate the effectiveness of our model, we provide additional rendering results in Figure 12, Figure 11 and Figure 9 of our supplementary. To demonstrate the temporal coherence of our model, we provide some videos by displaying included images in Figure 3 of the top comment PDF. We uniformly interpolate the time and randomly select a camera pose to render the scene reconstructed by Grid4D, and select the images per 5 frames (full 150 frames) to display. We will provide more videos on our GitHub repository in the final version.\n\n**Q4: Experimental results on other datasets such as Neu3D since D-NeRF is a toy dataset ?**\n\nA4: To further evaluate our model, we also conduct experimental comparisons on the real-world HyperNeRF and Neu3D datasets. The results demonstrate the effectiveness of our model on the real-world datasets.\n\n**Q5: Use average results of the time and space consumption instead of a range.**\n\nA5: We provide the detailed average metrics in the following table. We will add the table to our paper in the final version.\n\n|    Model     | Training Time | GPU Memory | FPS  | PSNR  |\n| :----------: | :-----------: | :--------: | :--: | :---: |\n|    4D-GS     |     20min     |    1GB     | 160  | 34.11 |\n|   DeformGS   |     33min     |   4.5GB    |  62  | 38.26 |\n|    SC-GS     |     75min     |   3.1GB    | 179  | 39.56 |\n| Grid4D(Ours) |     55min     |   4.0GB    | 153  | 39.86 |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mYJF23HYPn", "reviewer": "Reviewer_T422", "summary": "This paper proposes using Hash Encoding to model the Deformation Field for dynamic scenes. The authors first decompose 4D encoding into four 3D encodings to avoid the losses caused by the low-rank tensor assumption. They also introduce an attention module to decouple spatial and temporal features. Since explicit modeling can result in insufficiently smooth delta predictions for the deformation field, the authors incorporate a smooth loss to strongly regularize the outputs of Hash Encoding. Experiments on both synthetic and real-world datasets show substantial quality improvements compared to prior state-of-the-art methods. In addition, Grid4D shows a significant FPS improvement compared to Deformable-GS (without lightweight).", "questions": "A minor question: Lines 227-231 state that different scenes use different hyperparameters. I would like to know how significantly these different hyperparameters affect the scenes, as this greatly impacts the method's versatility.", "limitations": "Please refer to the Weakness section.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "I think the main strength of this paper is the authors' deep understanding of Deformable-based Gaussian splatting. To be compatible with the densification of vanilla Gaussian splatting, the deformation field needs to output delta (x or rotation). The key to the success of outputting delta lies in the sufficiently smooth output of the deformation field. This is why Deformable-GS [1] can outperform D-4DGS [2] significantly on datasets with accurate camera poses, since Hexplane is not as smooth as MLP. In addition, Hexplane not satisfying the low-rank tensor assumption in dynamic scenes is also a **potential reason**. The authors insightfully addressed the issue of MLP being overly smooth and explicit methods being insufficiently smooth. Therefore, they used Hash Encoding, which does not require the low-rank tensor assumption, to ensure high-frequency details, while employing smooth loss to ensure the foundation of the deformable-based Gaussian splatting.\n\n\n\nThe design of the deformation network and the experiments with strong baselines are compelling. Ablations are also appreciated.\n\n\n\n\n[1] Ziyi Yang, Xinyu Gao, Wen Zhou, Shaohui Jiao, Yuqing Zhang, and Xiaogang Jin. Deformable 3d gaussians for high-fidelity monocular dynamic scene reconstruction. arXiv preprint arXiv:2309.13101,2023.\n\n[2] Guanjun Wu, Taoran Yi, Jiemin Fang, Lingxi Xie, Xiaopeng Zhang, Wei Wei, Wenyu Liu, Qi Tian, and Xinggang Wang. 4d gaussian splatting for real-time dynamic scene rendering. arXiv preprint arXiv:2310.08528, 2023.", "weaknesses": "1. low-rank assumption -> **low-rank tensor assumption** is more accurate.\n2. I think the following papers should also be cited:\n   - Spacetime Gaussian Feature Splatting for Real-Time Dynamic View Synthesis by Zhan Li et al.\n   - 4D Gaussian Splatting: Towards Efficient Novel View Synthesis for Dynamic Scenes by Yuanxing Duan et al.\n3. Datasets:\n   - D-NeRF: I think `Lego` (wrong scene with inconsistent training and test set) should not appear in the table, or it should use the Deformable-GS [1] setup with the validation set as the test data. This can prevent misleading subsequent research.\n   - HyperNeRF: I do not find the results on HyperNeRF convincing because the camera poses in the HyperNeRF dataset are inaccurate. For example, in the `3D Printer` (Fig. 5), Grid4D is visibly clearer than TiNeuVox, but the metrics do not reflect this. Therefore, I suggest that the authors adopt `NeRF-DS` [3] dataset with more accurate camera poses for real-world scenes comparison.\n4. More difficult motion: the motions in the datasets used for experiments are mostly rigid or easy to explain. It would be interesting to see if the method is able to handle more difficult deformations, like non-rigid or large motion. Based on my understanding of the Deformation Field, it is challenging for it to handle large motions effectively.\n5. More comparisons: I wonder if the authors have considered SC-GS [4]. It could be better if the authors could compare their method with SC-GS. I believe the approach of SC-GS could be applied to Grid4D to achieve higher rendering quality and FPS.\n\n\n\n[3] Zhiwen Yan, Chen Li, and Gim Hee Lee. Nerf-ds: Neural radiance fields for dynamic specular objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8285–8295, 2023.\n\n[4] Yi-Hua Huang, Yang-Tian Sun, Ziyi Yang, Xiaoyang Lyu, Yan-Pei Cao, and Xiaojuan Qi. Sc-gs: Sparse controlled gaussian splatting for editable dynamic scenes. arXiv preprint arXiv:2312.14937, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the constructive feedback. We hope that our response below will address your concerns.\n\n**Q1: Citation of the two related works and writing problem.**\n\nA1: We will add the citation of the mentioned related works and change the presentation in the final version.\n\n**Q2: Should the metrics of the wrong scene 'Lego' be removed ?**\n\nA2: We will remove the metrics of the 'Lego' scene in the final version.\n\n**Q3: Use the NeRF-DS dataset with more accurate camera poses than the HyperNeRF dataset ?**\n\nA3: According to the original paper, we think that NeRF-DS dataset is designed for dynamic scenes with specular objects which is not suitable for general dynamic scene rendering. Therefore, to further demonstrate the improvements of Grid4D, we also choose the Neu3D dataset. Neu3D is captured by 21 cameras with fixed poses and has more accurate camera poses than HyperNeRF. Additionally, the scenes in the Neu3D dataset are larger and more challenging. \n\nWe report the comparison results with PSNR on the Neu3D[1] dataset in the following table, and our model has better performance than the state-of-the-art models. The qualitative results can be found in Figure 1 of the top comment PDF. We will add these experiments to our paper in the final version.\n\n|    Model     | Coffee Martini | Cook Spinach | Cut Beef  | Flame Salmon | Flame Steak | Sear Steak |   Mean    |\n| :----------: | :------------: | :----------: | :-------: | :----------: | :---------: | :--------: | :-------: |\n|    4D-GS     |     27.34      |    32.50     |   32.26   |    27.99     |    32.54    | **33.44**  |   31.01   |\n| Grid4D(Ours) |   **28.30**    |  **32.58**   | **33.22** |  **29.12**   |  **32.56**  |   33.16    | **31.49** |\n\n**Q4: How does Grid4D perform when handling difficult motions.**\n\nA4: The 'Hand', 'Peel Banana', 'Chocolate', 'Broom', and 'Teapot' scenes in HyperNeRF dataset have large and complex motions. As listed in Table 5, Figure 9, and Figure 10 of our paper, our methods have a much better rendering quality than the state-of-the-art models due to the discriminability improvement of explicit features. However, Grid4D still has several artifacts in the 'Broom' and 'Teapot' scenes, which might need further research.\n\n**Q5: Comparison to SC-GS.**\n\nA1: We conduct the comparison with SC-GS on the D-NeRF dataset. The comparison results with PSNR can be found in the following table, and the qualitative results can be found in Figure 2 of the top comment PDF. Notably, the original SC-GS rendering resolution is $400\\times 400$, lower than our model, so we changed the rendering resolution of SC-GS to $800\\times 800$ for fairness. Additionally, we remove the 'Lego' scene because of the incorrect test ground truth. We will add these experiments in the final version.\n\n|    Model     | Bouncing Balls | Hell Warrior |   Hook    | Jumping Jacks |  Mutant   |  Standup  |   Trex    | Mean      |\n| :----------: | :------------: | :----------: | :-------: | :-----------: | :-------: | :-------: | :-------: | --------- |\n|    SC-GS     |     41.59      |    42.19     |   38.79   |     39.34     |   43.43   | **46.72** |   39.53   | 41.65     |\n| Grid4D(Ours) |   **42.62**    |  **42.85**   | **38.89** |   **39.37**   | **43.94** |   46.28   | **40.01** | **42.00** |\n\nAccording to the results, Grid4D has better performance than SC-GS on average. The improvement of Grid4D might be because of the performance gap between implicit deformation fields (which SC-GS is based on) and Grid4D. SC-GS uses a full MLP-based implicit model to predict deformations, which has the over-smooth inherent property. Our proposed 4D decomposed hash encoder generates explicit features with high discriminability to represent the deformations more accurately.\n\nWe consider that SC-GS is an excellent work mainly focused on dynamic scene editing and rendering. Therefore, combining the advantages of SC-GS and Grid4D or applying Grid4D to SC-GS are probably great future works that might get better rendering quality.\n\n**Q6: What hyperparameters are set in different scenes ?**\n\nA6: In our experiments, the hyperparameters that need to change mainly include the resolution of the time dimension, the weight of smooth regularization term $\\lambda_r$, and the perturbation range $\\epsilon=(\\epsilon_x, \\epsilon_y, \\epsilon_z, \\epsilon_t)$. \n\nThe resolutions of the time dimension are set according to the frame number, usually about a half and a quarter of the frame number according to the Nyquist-Shannon Sampling Theorem. The resolution can be set automatically by the algorithm.\n\nThe weight of smooth regularization term $\\lambda_r=0.5$ for most scenes, $\\lambda_r=1.0$ for the 'vrig' part of HyperNeRF dataset. In most cases, setting $\\lambda_r=0.5$ could obtain reliable results. For better performance, increasing $\\lambda_r$ can help Grid4D render better when the scene primarily consists of rigid objects and simple motions. Similarly, decreasing $\\lambda_r$ can help Grid4D for modeling complex motions and non-rigid objects.\n\nFor D-NeRF dataset and the 'vrig' part of HyperNeRF dataset, we set the perturbation range into $\\epsilon\\in [-10^{-2}, 10^{-2}]$, For the 'interp' part of HyperNeRF dataset, we set the perturbation range into $\\epsilon\\in [-10^{-3}, 10^{-3}]$. In most cases, setting the range to $\\epsilon\\in [-10^{-3}, 10^{-3}]$ can obtain reliable results.\n\n[1] Li, Tianye, et al. \"Neural 3d video synthesis from multi-view video.\" *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*. 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank four reviewers for their constructive comments on how to improve our paper. We will provide individual responses below. The qualitative results on the real-world Neu3D dataset, the qualitative comparison to SC-GS, and a simple video exhibition can be found in the following PDF.

Meta-review: This submission received all positive final ratings after a heated discussion and strong rebuttal, and AC recommends acceptance. Please carefully address the concerns raised by the reviewer in the camera-ready version, especially regarding the new results on Neu3D, using the average performance in Table 7, including the reference suggested by the reviewer, and the video results. In addition, AC suggests the authors include and discuss NeRFPlayer in the revision, which also leverages HashGrid representation for dynamic field modeling.
Decision: Accept (poster)