Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction
Authors: Feng Wang
Zilong Chen
Guokang Wang
Yafei Song
Huaping Liu
Venue: NeurIPS 2023 spotlight (2023)
Keywords: NeRF
Dynamic Scenes
TL;DR: 4D video in a fast training and rendering speed.

Abstract: In this paper, we propose the Masked Space-Time Hash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size.Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene. We evaluate our method on extensive dynamic scenes. As a result, MSTH obtains consistently better results than previous state-of-the-art methods with only 20 minutes of training time and 130 MB of memory storage.

Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction
Feng Wang"!
Zilong Chen"!
Guokang Wang'
Yafei Song²
Huaping Liu
Beijing National Research Center for Information Science and Technology(BNRiis),
Department of Computer Science and Technology, Tsinghua University AAlibaba Group
vang-f200mails, tsinghua, edu. cn. hpliu@tsinghua. edu.
Abstract
In this paper, we propose the Masked Space-Time Hash encoding (MSTH), novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monoc- ular videos. Based on the observation that dynamic scenes often contain substantial static areas that result redundancy in storage and computations, MSTH repre- sents dynamic scene weighted combination of 3D hash encoding and 4D hash encoding. The weights for the two components are represented by a learnable mask which guided by uncertainty-bakee objective to reflect the spatial and temporal importance each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications static areas, making feasible to represent large number of space- since-time voxels by hash tables with small size. Besides, without the requirements fit the large numbers of temporally redundant features independently. our method is easier to optimize and converge rapidly with only twenty minutes of training for 300-frame dynamic scene. As result, MSTH obtains consistently better results than previous methods with only 20 minutes of training time and 130 MB of memory storage. Code available at https: / github. com cwm/maasea masked- spacetime- waateiiieeahhingneee hashing/msth
Introduction
Neural radiance fields (57
96
79
58
14
PSNR on Plenoptic Video
33
Ours
Ours
Emmersive Dataset
Plenoptic
Dataset
30
PSNR on Google Immerssie
have achieved great success in reconstructing static 3D scenes. The learned volumetric repre- sentations can produce photo-realistic render- ing for novel views. However, for dynamic scenes, the advancements lag behind that of static scenes due to the inherent complexities caused by the additional time dimension.
Ours
HyperReel
1
3
32.
K. Planes
HexPlane
72
Plame
10
31
28
HyperReel 1
NeRFPlayer
75
30
227
NcREPlayer
Specifically, reconstructing dynamic scenes re- quires more time, memory footprint, and addi- tional temporal consistency compared to static scenes. These factors have resulted in unsat- isfactory rendering qualities. Recently, many works 40 38 75
0.3
Training time (hour)
Figure I: Performance comparison. We compare the PSNR and training time two public benchmarks. Our method surpasses other methods non-trivial margin with only 20m of training. denotes the HexPlane 10) setting which removes the coffce-martini scene.
10 72 have made great progress in dynamic scene reconstruction, im- proving the efficiency and effectiveness of reconstructions. However, there is still considerable room
Equal contribution.
Corresponding Author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

DyNeRF 400
1344h
HexPlane 10 12h
Ours 20min
NcRFPlayer 75 6h
Heper 01 2.7h
Ours 20min
DyNeRF 1344h
HyperReel[]] 2.7h
Ours 20min
Figure 2: Qualitative comparisons to sfate-of-theaar methods 44
11 75 10
We visualize three
scenes: flame salmon, horse, and welder from Plenoptic Video dataset [40 and Google Immersive dataset (9). Some key patches are zoomed in for better inspection. Our method performs better in reconstructing details, such the stripes of the salmon, the facial features, and the splashing sparks.
for improvement in many aspects such as rendering quality, motion coherence, training and rendering speed, and memory usage.
This paper aims to develop method for reconstructing dynamic scenes with high rendering quality in a space- and time-efficient manner. To achieve this goal, we base our approach on the multi-resolution hash encoding (58) due to its efficiency and compactness for representing and reconstructing static scenes. However, directly applying a 4D multi-resolution hash table to represent a dynamic scene would require much larger hash table size than that for static scene due to the much more hash collisions caused by the additional time dimension. Our finding is that, for the ubiquitous existence of static areas in scene, storing the static points into a 4D time-dependent hash table will result in an information redundancy since each of them will occupy T hash table items with an identical value for T-faame scene. It will also lead high hash collision rate since narrows the capacity of the hash table, which negatively impacts the reconstruction quality. Therefore, for the points with low dynamics, we hope establish a mechanism reduce the frequent queries and updates the 4D hash table and automatically save their features into 3D hash table to avoid temporal redundancy.
From this observation, we propose Masked Space- Space-Time Hash encoding, which combines 3D hash encoder and a 4D hash encoder with a 3D learnable weight mask. In order to make the mask correlate with the dynamics of the corresponding positions, we adopt the Bayesian framework of Kendall and Gal (4 to estimate the uncertainty [34 [44 each point being static. The non-linear correlation [6] between the uncertainty and the learnable mask is maximized to make the mask reflect the dynamics. In this way, the static points indicated with a low uncertainty will have a low weight for the 4D hash table and a high weight for the 3D hash table, which prevents modifications to the dynamic representations. With the proposed masked space-time hash encoding, we can set the size of a 4D hash table the same 3D one without much loss of rendering qualities, making the representation highly compact. Besides, without the requirements to fit the large numbers of repeated features independently, our method easier optimize and converge rapidly in only twenty minutes. To validate the effectiveness of our method on scenes in more realistic settings with large areas of dynamic regions and more complex movements, we collect a synchronized multi-view video dataset with 6 challenging dynamic scenes, which will be publicly available. As result, the proposed masked space-time hash encoding achieves consistently better reconstruction metrics on two publicly available datasets consisting of 13 scenes, with only 20 minutes of training and 1330 MB of storage. Fig. [ and Fig. 2 show the quantitative and qualitative comparisons to other state-of-the-a methods. In summary, contributions are:

---

We propose Masked Space-tima Hash Encoding, a novel representation that decomposes the dynamic radiance fields into a weighted combination of 3D and 4D hash encoders.
The proposed method is validated on a wide range of dynamic scenes, surpassing previous state-of-the-an methods by non-trivial margins with only 20 minutes of training time and 130 MB of memory storage.
We propose a new synchronized multi-vies dataset with more challenging dynamic scenes, including scenes with many moving objects and scenes with complex and rapid movements.
2
Related Work
Neural Scene Representations for Static Scenes. Representing 3D scenes with neural networks has achieved remarkable success in recent years. NeRF [57] first proposes use neural networks to represent radiance fields, demonstrating the remarkable potential of this representation in conjunction with volume rendering techniques. The high-fidelity rendering has sparked a surge of interest in related areas. Numerous variants have emerged to address the inherent challenges of the original approach,
including improving rendering quality [3 5 35
52
90], handling sparse input [60 13
92 55
94 17).
lighting 88 76 62), editing [37 48 45 20 56), dealing with complicated [86] 25 95 36 and large
scenes 88 100, 54
[3] generalization 98 73
80
12
87
59
29
88 jointly optimizing with
[8] and speeding Our work draws inspiration from 58 and Mip-NeRF 360 [4]. In Instant- NGP, Müller et al. (58) proposes a novel data structure that leverages multi-resolution hash grids to efficiently encode the underlying voxel grid while addressing hash collisions using decoding MLP. Furthermore, the proposed method allows for fast and memory-efficient rendering of large-scale scenes. Similarly, Mip-NeRF 360 [4] presents a sample-efficient scheme for unbound scenes, which utilizes a small density field as a sample generator and parameterizes the unbounded scene with spherical contraction. In this paper, we build upon these contributions and propose novel approach that extends successful techniques and components to incorporate time dimension with minimal overhead. Our method is capable of representing dynamic scene with only 2-3x memory footprint than that of a static NeRF.
camera poses 43 89
7
30
78
103
16
accelerating training
96,
14,
79
93
up rendering
28
84
51
two recent contributions in the field, namely Dastant-NGP
Novel View Synthesis for Dynamic Scenes. Extending the neural radiance field to express dynamic scenes a natural yet challenging task that has been proven crucial to many downstream applications Many researchers focus on monocular dynamic novel view synthesis, which takes a monocular video as input and targets to reconstruct the underlying 4D information by modeling deformation explicitly [19 (2 41
39 66 77 31
91 68 |9] or implicitly [64 23 63
21
74
49 46 102 82 32
As an exemplary work, D-NeRF (68) models a vime-narying field through a deformation network that maps 4D coordinate into spatial point in canonical space. Despite the great outcomes achieved by research in this line, the applicability of these methods is restricted by the inherent nature of the underlying problem (33) A more practical way of reconstructing dynamic scenes is by employing multi-view synchronized videos [33 50 [04
85] 2 67]. DyNeRF [40] models dynamic scenes by exploiting a 6D plenoptic MLP with time queries and set of difference- based importance sampling strategies. The authors also contributed to the field by presenting a real-world dataset, which validated their proposed methodology and provided a valuable resource for future research endeavors. K-Planes (72) and HexPlane [10] speed up training by decomposing the underlying 4D radiance field into several low-dimensional planes, which substantially reduces the required memory footprint and computational complexity compared with explicit 4D voxel grid. HyperReel M breaks down the videos into keyframe-basee segments and predicts spatial offset towards the nearest keyframe. NeRFPlayer (75] and Mix Voxel [3] address the problem by decomposing 4D space according to corresponding temporal properties. The former separates the original space into three kinds of fields and applies different structures and training strategies, while the latter decouples static and dynamic voxels with a variational field which further facilitates high-quality reconstruction and fast rendering. Our method implicitly decomposes 3D space with learnable mask. This mask eliminates the requirement for manual determination of dynamic pixels and enables the acquisition of uncertainty information, facilitating high-quality reconstruction.
38 I 75 10
83
3

---

3 Methodology
Our objective is to reconstruct dynamic scenes from a collection of multi-view or monocular videos with compact representation while achieving fast training and rendering speeds. To this end, we propose Masked Space-Tima Hash encoding, which represents dynamic radiance field by weighted combination of a 3D and a 4D hash encoder. We employ uncertainty-guidee mask learning strategy to enable the mask to learn the dynamics. In the following, we will first introduce the preliminary, then the masked space-time hash encoding and uncertainty-guide mask learning, and finally the ray sampling method.
3.1 Preliminary
For neural radiance fields, the input is 3D space coordinate æ and direction d to enable the radiance field to represent the non-Lambertian effects. The output is the color c(æ; d) E IR and direction-irrelevant density (xx E R. Most existing methods encode the input coordinate with mapping function that maps the raw coordinate into Fourier features [57], voxels-based features 179, 96). hashing-basee features (58) factorized low-rank features [14]. In this work, mainly focus on the multi-resolution hash encoding due to its efficiency and compactness. Specifically, for an input point æ. the corresponding voxel index is computed through the scale of cach level, and a hash table index is computed by bit-wise XOR operation [61. There are I. levels of hash tables corresponding different grid sizes in geometric progressive manner. For each level, the feature for a continuous point tri- tri-linearly interpolated by the nearest grid points. The corresponding features in different levels are concatenated obtain the final encoding. For convenience, denote the output of the multi-resolution hash encoder for x as enc(æ).
After the multi-resolution hash encoder, density-decoding MLP da, is employed to obtain the density and an intermediate feature g such that: o(x). = Φe, (enc(z)). Then, the color is computed through color-decoding MLP thay: c(x) We, (x) , d) which is direction-dependent.
For rendering, points along a specific ray rss) + sd are sampled and the volumetric rendering 57 is applied to get the rendered color (r)):
c(r)= = f.¹ T(s) a(r(s)) c(r(s), d]ds, where ((a) exp (-f ((())(()))
(1)
squared error between the rendered color C(r) and the ground truth color C(r) is applied for pack-propagation
3.2 Masked Space-Time Hashing
For dynamic neural radiance field, the input a 4D space-time coordinate (ar, t). A straightforward method is to replace the 3D hash table with 4D one. However, this simple replacement will result in a high hash collision rate due to the enormous volume of hash queries and modifications brought by the additional time dimension. For instance, in dynamic scene comprising T frames, the hash collision rate T times higher than a static scene, leading to a degradation the reconstruction performance. Enlarging the size of the hash table will cause an unbearable model size and be difficult to scale up with the frame numbers.
To solve this problem, we propose the masked space-time hash encoding, which incorporates a 3D multi-resolltion hash mapping hap, a 4D multi-resolution hash mapping hap. and learnable mask encoding m.. The final encoding function is formulated as follows:
enc(a x, t) = m(x) hap (x) + (1 mm()) hap( t), where m(x) = sigmoid(m((z)).
The learnable mask m can be represented by 4D hash table, the sizes of time steps also adopt geometric growing multi-resolution scheme, which reasonable due to the natural hierarchical properties of motions in time scales.
hash table or 3D voxel grid. For the
After obtaining the space-time encoding, the density- and color-decoding MLPs are applied to obtain the final outputs:
t), g(x,t) t) de, (enc(æ, t)), c(xx, d, ve, (g(x, d).
(3)

---

(a) Origin Image
(b) Mask without Uncertainty
(c) Mask with Uncertainty
Figure 3: We compared the learned masks m by visualizing them using volumetric rendering. As shown in the figures above, we observed that the mask learned with uncertainty is cleaner and tends to have a binarized value, which helps avoid the mixture of both tables.
Then the volumetric rendering is applied to each ray at each time step, and squared error is employed as the loss function:
Ö(r,1)= = S( T(s,1) ((((s))) d, L, - -.=1110000-000 Ö(r, 01B] r.
(4)
Reduce Hash Collisions. The intuition behind the masked modeling lies in the fact that many parts in the dynamic radiance fields are time-invariant, such as the static objects, the background, etc. These static parts can be well reconstructed from the 3D hash table with large m. For these static points, storing their features in the 4D hash table will occupy a large number of storage, largely increasing the hash collision rate. For these time-invariant parts, the 3D hash encoder can be sufficient to reconstruct, and these static parts will not modify the 4D hash table significantly when (1 - m (x)) small. In this way, the 4D hash table stores the properties of those points which are really dynamic, and those dynamic features in the 4D hash table will be protected by the mask term to suppress the gradients from static points.
Another advantage of using masked space-time hash encoding is to accelerate the fixed view-port rendering. Instead of rendering a novel view video frame-by- by-mme,,, adopt an incremental way of using the mask to avoid redundant computations. Specifically, for a T-frame scene, we first render the initial frame Vo as usual. The obtained mask is used to filter the static parts, and we only render the dynamic parts of other frames with the dynamic weight (1 mm))) > (1 -e), where e is a pppree.rrrettr... In this way, except for the initial frame, we only require to render the dynamic part, improving the rendering fps from 1.4 to 15, without loss of rendering quality.
A Acclerate Rendering.
The key to the masked space-time hash encoding is that m(æ) can reflect the dynamics of the point. To this end, we design an uncertainty- the 3D hash uncertainty u(x:) and the mask m(x). We will elaborate this strategy in the next part.
mask learning strategy to connect the relation between
3.3 Uncertainty-guided Mask Learning
To make the mask m(x)) well reflect the dynamics of the corresponding point, we design an uncertainty branch in our model to estimate the uncertainty of point being static, which is good indicator for the dynamies of the point. To this end, the uncertainty branch is required to reconstruct the dynamic scenes only using the 3D hash table with the input-dependent uncertainties, ignoring the time input. In this way, the dynamic points are regarded as the inherent noise since its supervision are inherently inconsistent (different time step describes different geometries while the uncertainty model is time-agnostic). The inherent noise of dynamic points will lead to a high uncertainty. We adopt the Bayesian learning framework of Kendall and Gal 34 model the heteroscedastic aleatoric uncertainty for each point.
Specifically, we construct an uncertainty field with voxel grid representation. u(x:) denotes the uncertainty of space point æ. We denote the raw output of the uncertainty voxel-grid as ū, and a soft-plus is used as the activation: u(x)) = Um + +og (1 + exp(ū(x))) where umm is hyper-parameter for shifting the uncertainty values [54]. For each ray, the ray-level uncertainty U(r) is calculated through volumetric rendering [53]:
U(r)= J' ((s)-o(r(e) u(((s))ds.
(5)
5

---

Besides, the color and density estimated by the 3D hash table are:
(z), go (x) = Φo, (hao( ca d) = tee (g. (z).d))
(6)
The rendered color of this branch is (r(r) by applying the volumetric rendering. After that, the uncertainty-batt based loss for ray r is defined as:
La E r, 2U( [C(r,t) (.()) +
Note that the uncertainty branch is time-agnostir i.c., the predicted color C(r) is not time-dependent, which is important to make the uncertainty relevant to dynamics. In this way, the uncertainty for each point u(x) can be estimated through the above loss function. For the dynamic points, the uncertainty is inherently large because of the ambiguous and inconsistent geometries caused by the inconsistent pixel color supervision. In this perspective, the uncertainty well reflect the dynamics of each point and could guide the mask values.
Bridging uncertainty with mask. Though we find the correlation between the mask m and uncer- tainty u, it is not trivial to connect them. First, m and u are in different value ranges, m((r) E [0, 1] while u(x) E [0, +xx). Second, the distributions of m and u are very different, and the relations between them are non-linear. Imposing a hard relationship between them will impact the training of their specific branches. We instead maximize the mutual information between the two random variables m and u to maximize the nonlinear correlation between them. Although MI can not measure whether the correlation is negative, Eq. (7) can guarantee this. The mutual information [m, u) describes the decrease of uncertainty in m given u: I(m, u) == (m) - Hmm lu), where H is the Shannon entropy. To estimate the mutual information between and u, we adopt the neural estimator in [6 to approximate the mutual information [m,, u) as:
(m,, u) = sup Ep.... [Te] - log( &F. [)) Ag
(8)
Pm,. is the joint probability distribution of m and u, and Pm @ Pu is the product of marginals. Te is neural network with parameters 0 O. We choose small MLP with two hidden layers to represent To and draw samples from the joint distribution and the product marginals to compute the empirical estimation of the expectation. By maximizing the estimated mutual information, we build the correlation between m and u. At last, the overall learning objective of MSTH is the combination of the above losses:
L= + A L. 7 Ie (m,, u),
where À and y are two hyper-paramettrs. For rendering a novel view, the uncertainty branches and the mutual information network are disabled.
In practice, the model learn a reasonable mask m even without the uncertainty guidance. However, this will make the learned mask very noisy and tend to learn a "middle value" in [0, 1], which will make the static points still have a relatively large dynamic weight modify the 4D table. Fig. B visualize the learned mask with or without uncertainty guidance. With uncertainty guidance, the mask will tend to be binarized and the static parts are with very low dynamic weight. For the mutual information constraint, we find it will make the distribution of m towards Bernoulli distribution which is helpful for reducing hash collision and accelerating rendering speed. Without the constraint, the model tends to learn more from the 3D hash table only even for the dynamic point. This will make the failing captures of some dynamic voxels with transient changes. As result, the uncertainty guidance will help learn a finer detail, which will be shown in the ablation part.
3.4
Ray Sampling
For natural video, a significant portion of the scene is usually static or exhibits only minor changes in radiance at a specific time across the entire video. Uniformly sampling the space-time rays causes an imbalance between the static and dynamic observations. Therefore, we sample the space-time queries according to the quantification of dynamics. Specifically, we sample a space-time ray (r, t) according to pre-computed probability P(r, 1). We decompose the sampling process into two sub-process: spatial sampling and temporal sampling. Formally, we decompose P(r, () P(r, () = P(()))((r), which forms a simple Markov process that first sample the space ray according to P(r) then sample the time step by ((t]r).

---

Plenoptic Video Dataset
Immersive Dataset
Campus Dataset
Figure 4: Qualitative results on different datasets of our method. We visualize four novel views in each column, three for RGB and one for depth. For other scenes, we visualize them in the Appendix, and the full spiral videos are presented in our supplemental material.
In practice, we build P(r) as the softmax of temporal standard deviation std(r) of the corresponding pixel intensities {C(r, t)::
P(r) x exp (std(r)/TT))
(10)
where Ti is the temperature parameter. We set a higher temperature for a softer probability distribution (27]. For temporal sampling, we follow 42 compute the weight according to the residual difference of its color to the global median value across time ((r) with temperature T2:
P(t|r) x exp (IC(r, f) - ((r))/m).
(11)
3.5 Implementation Details
To improve the sampling efficiency, we utilize the proposal sampler 4 that models density at coarse granularity with much smaller field, thereby generating more accurate samples that align with the actual density distribution with minor overhead. In our implementation, we use one layer of proposal net to sample 128 points. For the mask, we use a non-hash voxel grid with 128 spatial resolution. To encourage the separation of the static and the dynamic, we utilize a mask loss that aims at generating sparse dynamic voxels by constraining the mask to be close at 1. We also adopt distortion loss 4 with Aaist 2e 2. For uncertainty loss, we set 3e 4 and À 3e 5. We find a small coefficient of mutual information estimator y will have a large impact on the gradients, which is consistent with the observations of (6). For the hash grid, we implement a CUDA extension based on PyTorch [65] to support the rectangular voxel grid required by the proposed method. For the complete experiment setting and model pareee-aaraeeeess please refer to the Appendix.
4 Experiments
4.1 Dataset
For validating the performances of the proposed method, we conduct experiments on two public datasets and our collected dataset: (1) The Plenoptic Video Dataset [4], which consists of 6 publicly accessible scenes: roffee-martini. hame-salmon, sear-steak. Each scene contains 19 videos with different camera views. The dataset contains some challenging scenes, including objects with topology changes, objects with volumetric effects, various lighting conditions, etc. (2) Google Immersive Dataset [9] The Google Immersive dataset contains light field videos of different indoor and outdoor environments captured by a time-synchronized 46-fishee 46-fisheye camera array. We use the same 7 scenes (Welder, Flames, Truck, Exhibit, Alexa Meade
-pinach,,,,, cut-roasted-beet beef, flame--teak, and
7

---

Table 1: Quantitative results Plenoptic Video dataset 40
We report the average metries and compare them with other state-of-the-art methods. Our method achieves non-trivial performance improvements on all metrics. denotes the DyNeRF setting which only reports results the flame-salmon scene. i denotes the HexPlane setting that removes the coffee-martini scene to calculate average metries. We report the per-scene metrics in the Appendix.
Model
PSNR1
D-SSIMI
LPIPS|
Training Time.
Rendering FPS t
Storage +
DyNeRF
40
29.58
0,020
0.099
1344
<0.01
28MB 135MB
Ours *
29.92
0,020
0.063
20 min
15
NeREPlayer
75
30.69
0,034
0.111
6 h
0.05
HyperReel
31.10
0,036
0.096
9
2.0
360MB
Mix Voxels
83
30.80
0,020
0.126
min
16.7
500MB
K- ElPaanes
72
31.63
0,018
108 min
Ours
32.37
0.015
0.056
20 min
15
135MB
HexPlane
10
31.705
0,014
0.075
12
200MB
Ours#
33.099
0.013
0.051
20 min
15
135MB
Exihibit, Face Paint 1, Face Paint 2) as NeRFPlayer [75] for evaluation. (3) To validate the robustness of our method on more complex in-the-wild scenarios, we collect six time-synchronized multi-view videos including more realistic observations such as pedestrians, moving cars, and grasses with people playing. We named the collected dataset as Campus Dataset. The Campus dataset is much more difficult than the above two curated ones in the movement complexities and dynamic areas. For detail on the dataset, please see Appendix. For the above three multi-view datasets, follow the experiment setting in [40] that employs 18 views for training and view for evaluation. To quantitatively evaluate the rendering quality novel views, measure PSNR, DSSIM, and LPIPS 1101 on the test views. We follow the setting of (40] to evaluate our model frame by frame. Our method is also applicable monocular videos. To validate the reconstruction quality with monocular input, we conduct experiments on D-NeRF [68] dataset, which contains eight videos of varying duration, ranging from 50 frames to 200 frames per video. There is only one single training image
for each time step. For evaluation, we follow the common setting from 68 72
4.2 Results
Multi-view Dynamic Scenes. We reconstruct the dynamic scenes from multi-view sycchrynchro observations on the two public multi-view video datasets and our collected Campus dataset. The quantitative results and comparisons are presented in Tab. and Tab. 2 For the Plenoptic Video dataset, our method surpasses previous state-of-the-ar methods by a non-trivial margin, with a 0.7 to 1.4 PSNR gains, and > 30% improvements the perceptual metric LPIPS. For training, cost at most 1/5 training time of other methods, achieving a speedup of 5 36 compared with other fast reconstruction algorithms and 4000 speedup compared with DyNeRF [40]. Besides, also keep a compact model size with only 135MB of memory occupancy, showing the advantages of the masking strategy which can avoid large number of hash collisions and make the dynamic hash table size smaller enough. For the Google Immersive dataset, Tab. [2 alsoo shows consistently non-trivin -tivial improvements in both training efficiency and reconstruction quality.
Fig. 2 visualizes the qualitative comparisons of our method to other state-of-the-ar methods, from which we can observe that our method can capture finer motion details than other methods. For example, MSTH can well reconstruct the fire gun with distinct boundaries and specular reflection, the mark of the hat, and the stripe of the salmon. The Splashing sparks can also be accurately captured. We provide the representative novel-view rendering results of our method in Fig.
Monocular Dynamic Scenes. We present the quantitative and qualitative results of our proposed method on the D-NeRF dataset Tab. 3 and Fig. [5] accompanied by a comparative analysis with related approaches. MSTH outperforms all other methods in terms of SSIM and LPIPS, including methods that focus on dynamic scenes (K-Planes [72) and HexPlane I00 and ToxeuVox, the current state-of-the-ant method for monocular dynamic novel view synthesis. The proposed MSTH achieves superior results without any assumption about the underlying field which demonstrates that our method could be casily extended to accommodate scenarios with varying complexity and dynamics.

---

Figure 5: Qualitative results on D-NeRF dataset [68]. The visualized images are rendered from test views. Complete per-scene multi-view images are shown in the Appendix.
Table 2:
Quantitive result on Google Immersive dataset [9] and the proposed Campus dataset. Check out the Appendix for detailed comparison and visualization.
Table 3: Quantitative comparison with re-
lated work on D-NeRF dataset [68] Per-
scene metrics are shown in the Appendix.
Model
PSNR
SSIM
LPIPS
Training Time +
Model
PSNR
SSIM
LPIPS]
Google Immersive
Video Dataset
T-NeRF 68
29.51
0.95
11088
NeRFPluyer 75
26.6
0.870
0.1931
D-NeRF
68
30.5
0.95
0.07
HyperReel
28.8
0.874
0.193
2.7 hrs
TiNeuVox-B
32.67
0.97
0.04
Ours
29.6
0.950
0.0929
20 min
HexPlane
10
31.04
0.97
0.04
K- PlPalans
30.84
0.96
Campus Dataser Ours
20.9
0.722
0.241
Ours
31.34
0.98
0.02
4.3 Ablation Study
Ablation on the Masked Hash Encoding. To evaluate the effect of decomposing 4D dynamic radiance field into the proposed masked space-time hash encoding, we propose two variants as comparisons: (1) A pure 4D hash encoding. (2) A simple decomposition which is addition of 3D hash table and a 4D hash table. Fig. [6] (a) and (b) visualize the qualitative comparisons. From Fig. (a) we can observe that only using the 4D hash table will generate blurry rendering with 3 PSNR drop. With a simple addition method, the reconstruction quality is improved compared with only 4D hash table, while the dynamic regions not captured well. Fig. (b) and (c) show the comparisons of MSTH with the addition. MSTH improves the LPIPS by 20% compared with the addition method.
Ablation on the uncertainty-based guid- ance. We visualize the learned mask with or without uncertainty-batt guid- ance, Fig. 3 shows the comparison. The learned mask with uncertainty loss will dis- tinguish the static and dynamic parts more clearly, with less noisy judgment, leading to assertive distribution which is beneficial to avoid hash collisions. Fig. (d) visualize the qualitative compar- isons. MSTH with uncertainty loss per- forms better the motion details.
the mask,
(b)
(c)
(d)
Figure 6: Qualitative comparisons for different abla- tions. Zoom in for better inspection.
5 Limitations and Conclusion
Limitations. MSTH tends to generate unsatisfying results when detailed dynamic information is insufficient, especially in monocular dynamic scenes, since relies solely on the input images. When the input is blurred, occluded, or incomplete, the method may struggle to infer the motion information, leading to artifacts such as ghosting, flickering, or misalignment, which may degrade the visual quality. Further research is needed to develop advanced methods that handle complex scenes, motion dynamics and integrate multiple information sources to enhance synthesis accuracy and robustness.
Conclusion. In this paper, we propose new method to reconstruct dynamic 3D scenes in time- and space-efficient way. We decouple the representations of the dynamic radiance field into time- invariant 3D hash encoding and time-varie variant 4D hash encoding. With an uncertainty -giided mask as weight, our algorithm can avoid a large number of hash collisions brought about by the additional time dimension. We conduct extensive experiments to validate our method and achieve of the-art performances with only 20 minutes of training. Besides, we collect a complex in-the-wile multi-view video dataset for evaluating the robustness of our approach more realistic dynamic

---

scenes, including many daily activities. We hope our method can serve as fast and lightweight baseline to reconstruct dynamic 3D scenes, which may derive many valuable applications such as interactively free-viewpoint control for movies, cinematic effects, novel view replays for sporting events, and other VR/AR applications.
Acknowledgement
This work supported in part by the National Natural Science Fund for Distinguished Young Scholars under Grant 62025304, Thank Xueping Shi for giving helpful discussion.
References
[1] B. Attal, J.-B. Huang, C. Richardt, M. Zollhoefer, J. Kopf, M. O'Toole, and C. Kim. Hyperreel: High- fidelity 6-dof video with ray-conditioned sampling. arXiv preprint wrXiv:2301.02255 2023.
[2] A. Bansal, M. Vo, Y. Sheikh, D. Ramanan, and Narasimhan. visualization dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages (175, 2020.
[3] J. T. Barron, B. Mildenhall, M. Tancik, Hedman, R. Brualla, and P. Srinivasan. Mip-nerf:
A multiscale representation for anti- anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855-5864, 2021.
[4] J. T. Barron, Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 5470-5479, 2022.
[5] J. T. Barron, B. Mildenhall, D. Verbin, P. Srinivasan, and Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv, 2023.
[6] M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y. Bengio, A. Courville, and R. D. Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801 100002000..0062,, 2018.
[7] W. Bian, Z. Wang. K. Li, J. Bian, and V. A. Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023.
[8] M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lenseh. NeRD: Neural reflectance decomposition from image collections. https://ariioorr/abs//02222
2020.
[9] M. Broxton, J. Flynn, R. Overbeck, D. Erickson, P. Hedman, M. Duvall, J. Dourgarian, Busch,
M. Whalen, and P. Debevec. Immersive light field video with layered mesh representation. ACM Transactions on Graphics (TOG), (9,4) 2020.
[10] A. Cao and J. Johnson. Hexplane: fast representation for dynamic scenes. rrXiv:2301.0663 2023.
arXiv preprint
[11] J. Cao, H. Wang, P. Chemerys, V. Shakhrai, J. Hu, Y. Fu, D. Makoviichuk, S. Tulyakov, and J. Ren. Real-time neural light field on mobile devices. arXiv preprint arXiv:2212.08055. 2022.
[12] Chan, M. Monteiro, P. Kellnhofer, Wu, and G. Wetzstein. pi-GAN: Periodic implicit generative 2020.
adversarial networks for 3D-aware image synthesis. https://arai..org/a////02222222
[13] A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang. J. Yu, and H. Su. Mysnerf: Fast generalizable radiance field reconstruction from multi-view stereo. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14124-1413
2021.
[14] A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2233.09512, 2022.
[15] X. Chen, Q. Zhang, X. Li, Y. Chen, F. Ying, X. Wang, and J. Wang. Hallucinated neural radiance fields in the wild, 2021.
[16] Chen, X. Chen, X. Wang, Q. Zhang. Y. Guo, Y. Shan, and Wang. Local-to-global registration for bundle-adjusting neural radiance fields. arXiv preprint arXiv:2211.11555 2022.
[17] J. Chibane, A. Bansal, Lazova, and G. Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, jun 2021.
[18] K. Deng, A. Liu, J.-Y. Zhu, and D. Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arXiv preprint arXiv:2107.0279. 2021.
[19] Y. Du, Y. Zhang, H.-X. Yu. J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing, In Proceedings of the IEEE/CVF International Conference Computer Vision, 2021.
10

---

[20] Z. Fan, Y. Jiang. P. Wang. X. Gong, D. Xu, and Z. Wang. Unified implicit neural stylization. In Computer Visio-ECCV 2022: 17th European Conference, Tel Aviy, Israel, October 23-27, 2022, Proceedings, Part XV. pages 636-654, Springer, 2022.
[21] J. Fang. T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers, 2022.
[22] C. Gao, A. Saraf, J. Kopf. and J.-B. Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712-5721, 2021.
[23] H. Gao. R. Li, Tulsiani, B. Russell, and A. Kanazawa, Dynamic novel-view synthesis: A reality check.
In NeurlPS, 2022.
[24] S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision,
pages 14346-14355, 2021.
[25] Y.-C. Guo, D. Kang, L. Bao, He,, and S.-H. Zhang. Nerfren: Neural radiance fields with reflections. In Pruceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages
18409-1841 18418, June 2022.
[26] P. Hedman, P. Srinivasan, B. Mildenhall, J. T. Barron, and Debevec. Baking neural radiance fields for real-time view synthesis. ICCV, 2021.
[27] G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint MXXi::553.0253/ 2015.
[28] T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition ( (CVPR), pages 12902-12911, June 2022.
[29] W. Jang and L. Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 12949- -2958, October 2021.
[30] Y. Jeong. S. Ahn, A. A. Christopehr Choy, M. Cho, and J. Park. Self-calibrating neural radiance fields. In ICCV, 2021.
[31] Jiakai, L. Xinhang, Y. Xinyi, Z. Fuqiang. Z. Yanshun, W. Minye, Yingliang, X. Lan, and Y. Jingyi. Editable free-viewpoint video using a layered neural representation. In ACM SIGGRAPH, 2021.
[32] Y. Jiang, P. Hedman. B. Mildenhall, D. Xu, J. T. Barron, Z. Wang. and T. Xue. Alignerf: High-fidelity neural radiance fields via alignment- aware training. arXiv preprint arXiv:2211.0966 0.0682, 2022.
[33] T. Kanade, P. Rander, and P. Narayanan. Virtualized reality: Constructing virtual worlds from real scenes. IEEE multimedia, 4(1):34 1997.
[34] A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems, 30, 2017.
[35] Kerbl, G. Kopanas. Leimkühhler, and Drettakis. 3d gaussian splatting real-time radiance field rendering. ACM Transactions on Graphics, 42(4), July 2023. URL http:://reppp sam inria. fr/ fungraph/3er Sumggraph/dd-ggussiin--ppo
[36] Klenk, Koestler. D. Scaramuzza, and D. Cremers. E-nerf: Neural radiance fields from a moving event camera. IEEE Robotics and Automation Letters, 2023.
[37] Kobayashi, E. Matsumoto, and Sitzmann. Decomposing nerf for editing via feature field distillation.
Advances Neural Information Processing Systems, volume 35, 2022. URL/jttpss//arxiv org/ pdf /220/2005.40 15585. pdf
[38] L. Z.. Shen, Z. Wang. L. Shen, and P. Tan. Streaming radiance fields for 3d video synthesis. arXiv preprint arXiv:2210. 14831,, 2022.
[39] R. Li, J. Tanke, M. Vo, M. Zollhofer, J. Gall, A. Kanazawa, and C. Lassner. Tava: Template-free animatable volumetric actors. 2022.
[40] T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al. Neural 3d video synthesis from mieti-view video. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5521-5531, 2022.
[41] Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), 2021.
[42] Z. Li. S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498 6508, 2021.
11

---

[43] C.-H. Lin, W.-C. Ma. A. Torralba, and S. Lucey. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV), 2021.
[44] D. B. Lindell, J. N. Martel, and G. Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages
2021.
[45] H.-K. Liu, L-C. Shen, and B.-Y. Chen. Nerf-in: Free-form nerf inpainting with rgb-d priors. arxiv preprint arXiv:2206.0499 04901, 2022.
[46] J.-W. Liu, Y.P. Cao, W. Mao, W. Zhang, D. J. Zhang, Keppo, Y. Shan, X. Qie, and M. Z. Shou. Devef: Fast deformable voxel radiance fields for dynamic scenes. arXiv preprint arXiv:2205.1572 2022.
[47] L.. Liu, J. Gu, K. Zaw Lin, T.-.. Chua, and C. Theobalt. Neural sparse voxel fields. Advances Neural Information Processing Systems, 33:15651- 2020.
[48] S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell. Editing conditional radiance fields. 2021.
[49] Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y-Y. Chuang, J. Kopf, and J.-B. Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and
Pattern Recognition, 2023.
[50] S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.0775// 2019.
[51] Lombardi, Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and Saragih. Mixture of volumetric primitives for efficient neural rendering, 2021.
[52] L. Ma, X. Li, J. Liao, Q. Zhang, X. Wang, J. Wang, and P. V. Sander. Deblur-nerf: Neural radiance fields from blurry images. arXiv preprint arXiv:21 11.22111,4492,, 2021.
[53] R. Martin-Brualla, N. Radwan, M. S. Sujjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7210-7219, 2021.
[54] R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR, 2021.
[55] Q. Meng, A. Chen, H. Luo, M. Wu, H. Su, L. Xu, X. He, and Yu. Gnerf: Gan-based neural radiance field without posed camera. arXiv preprint arXiv:2103, 15606, 2021.
[56] B. Mildenhall, P. Hedman, R. Martin- Martin-Brualla, P. P. Srinivasan, and T. Barron. NeRF in the dark: High d dynamic range view synthesis from noisy raw images. arXiv, 2021.
[57] B. Mildenhall, P. P. Srinivasan, M. Tancik, T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1)999-106 2021.
[58] T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphies primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.0500 2022.
[59] M. Niemeyer and A. Geiger. Campari: Camera-aware decomposed generative neural radiance fields,
1021.
[60] M. Niemeyer, J. T. Barron, B. Mildenhall, M. M. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2022.
[61] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Trans. Graph., 32(6); 1-99:116::11,,,1111-111111111111 2013. doi: 10.1 145/2508363..508877 URL.
https://doo wawloi.orggoiie orgoro/////// 10. 0144/2508836 2508374
[62] X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A shading-guidess generative implicit model for shape- accurate 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[63] K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla, Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865 5874555772242 2021.
[64] K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and M. Seitz. Hypernerf: A dimenssoal representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106. 13228, 2021.
[65] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Lin, N. Gimelshein.
L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang. Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner. L. Fang. J. Bai, and S. Chintala. Pytorch: An imperative style, high perforrefree deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché- Buc. E. B. Fox, and
12

---

R. Gametl, editors, Advances in Neural Information Processing Systems 32: Annual Conference on
Neural Information Processing Systems 2019, NewrlPS 2019, December 8-14, 2019, Vancouver, BC. Canada, pages 8024-8035. 2019. URL Abstract.htam
seerrpps.cc/ppprr2011.. 2019//////
bobcc228feee792222rrass
[66] S. Peng, Y. Zhang. Y. Xu, Q. Wang. Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR, 2021.
[67] S. Peng. Y. Yan, Q Shuai,, H. Bao, and X. Zhou. Representing volumetric videos as dynamic mlp maps.
In CVPR, 2023.
[68] A. Pumarola, E. Corona, G. Pons- Poll-Moll, and F. Moreno -ogueer D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2021.
[69] D. Rebain, W. Jiang, S. Yazdani, K. Li, K. M. Yi, and A. Tagliasacchi. Derf: Decomposed radiance fields. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14153-14161, 2021.
[70] C. Reiser, S. Peng. Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2021.
[71] C. Reiser, R. Szeliski,, D. Verbin, P. P. Srinivasan, B. Mildenhall, A. Geiger, J. T. Barron, and P. Hedman. Merf: Memory-effiiient radiance fields for real-time view synthesis in unbounded scenes, arXiv preprint arXiv:2302. 12249, 2023.
[72] Sara Fridovich-Kei and Giacomo Meanti, F. R. Warburg, B. Recht, and A. Kanazawa, K-planes: Explicit radiance fields in space, time, and appearance. In CVPR, 2023.
[73] K. Schwarz, Y. Liao, M. Niemeyer. and A. Geiger. Graf: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS), volume 33, 2020.
[74] Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2023.
[75] L. Song, A. Chen, Z. Li, Z. Chen, L. Chen, J. Yuan, Y. Xu, and A. Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. arXiv preprint arXiv:2210.15947 2022.
[76] P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. htpps//arri.orr/a///01122..83
2020.
[77] S.-Y. Su, F. Yu, M. Zollhöfer, and H. Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In Advances in Neural Information Processing Systems, 2021.
[78] Sucar, S. Liu, Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-time. In arXiv preprini arXiv:2103. 123522, 2021.
[79] C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5459-5469, 2022.
[80] M. Tancik, B. Mildenhall, T. Wang. D. Schmidt, P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-based neural representations. https:/aarrisorn/dab222220002200 2020.
[81] M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan. J. T. Barron, and H. Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8248-82558 2022.
[82] E. Tretschk, A. Tewari, V. Golyanik, M. Zollhäfer, C. Lassner, and C. Theohalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings the IEEE/CVF International Conference on Computer Vision, pages 12959-12970, 2021.
[83] F. Wang. S. Tan, X. Li, Z. Tian, and H. Liu, Mixed neural voxels for fast multi-view vidco synthesis. arXiv preprint arXiv:2212.0011 2022.
[84] H. Wang, Ren, 7. Huang, K. Olszewski, M. Chai, Y. Fu, and S. Tulyakov. R21: Distilling neural radiance field to neural light field for efficient novel view synthesis. In F.CVV, 2022.
[85] L. Wang. J. Zhang, X. Liu, F. Zhao, Y. Zhang, Y. Zhang, M. Wu, J. Yu, and L. Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13524-13534, 2022.
[86] P. Wang, Y. Liu. Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, and W. Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR, 2023.
13

---

[87] Q. Wang. Z. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualle N. Suavely, and T. Funkhouser. Ibrnet: Learning multi-view image-based rendering. arXiv preprint 17X1v22102.3300.
2021.
[88] T. Wang, B. Zhang, T. Zhang. S. Gu, J. Bao, T. Baltrusaitis, Shen, D. Chen, F. Wen, Q. Chen, et al. Rodin:
A generative model for sculpting 3d digital avatars using diffusion. arXiv preprint 4rX1::2212.06135. 2022.
[89] Z. Wang, S. Wu, W. Xic, M. Chen, and V. A. Prisacariu. NeRF--: Neural radiance fields without known camera parameters. arXiv preprint ar Xiv: 2xx:::102.000000 2102007064 2021.
[90] Wang, L. Li, Z. Shen, L. Shen, and L. Bo. 4k-nerf: High fidelity neural radiance fields at ultra high resolutions. Xrxiv preprint arXiv:2212.04001 2022.
[91] W. Xian, J.-B. Huang, J. Kopf, and C. Kim. Space-time neural irradiance fields for free-viewpoint video. Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pages 9421-9431, 2021.
[92] D. Xu, Y. Jiang. P. Wang. Z. Fan, H. Shi, and Z. Wang. Sinnerf: Training neural radiance fields complex scenes from single image. 2022.
[93] L. Yariv, P. Hedman, C. Reiser, D. Verbin, P. P. Srinivasan, R. Szeliski, J. Barron, and B. Mildenhall. Bakedsdf: Meshing neural sdís for time view synthesis. arXiv preprint UrXiv:2302.14833 2023.
[94] L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez. P. Isola, and T.-Y. Lin. iNeRF: Inverting neural radiance fields for pose estimation. http://auriixorr///22221... 058777777700707... 2020.
[95] Z.-X. Yin, Qiu, M.-M. Cheng, and B. Ren. Multi-space neural radiance fields. 2023.
[96] A. Yu, S. Fridovich-Kede M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields
without neural networks. arXiv preprint vrXiv:2112.05133 2021.
[97] A. Yu, R. Li, M. Tancik, H. Li, R. Ng. and A. Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5752-5761. 2021.
[98] A. Yu, V. Ye, M. Tancik, and A. Kanazawa, pixeINeRF: Neural radiance fields from one or few images. In CVPR, 2021.
[99] W. Yuan, Lv, T. Schmidt, and S. Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattenn Recognition, pages 13144-13152, 2021.
[100] K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arX1v:2010.07492, 2020.
[101] R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586-595. 2018
[102] F. Zhao, W. Yang, J. Zhang. P. Lin, Y. Zhang. J. Yu, and L. Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7743-7753 June 2022.
[103] Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice-slam: Neural implicit scalable encoding for slam, 2021.
[104] C. L. Zitnick, S. B. Kang, M. Uytenndacle, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered reprosentation. ACM transactions on graphics (TOG), 23(3)6000-688, 2004.
14


Reviews:
[{"review_id": "l7RZFcGjxy", "reviewer": "Reviewer_r2SF", "summary": "This paper tries to address the challenge of efficiently representing 3D dynamic scenes. It proposes a decoupled representation that uses separate neural implicit representations for dynamic and static 3D points. This approach can reduce hash collision and save the storage of the multi-level hash feature grids.", "questions": "In Fig.6, what is the meaning of each row?", "limitations": "The proposed method is most effective for scenes with a significant amount of static parts. For scenes with a small amount of static parts, the storage savings will be less significant.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. An automatic learning procedure to detect dynamic points in the 3D scene. This is achieved by integrating Bayesian learning to treat dynamic points as noise and reduce their weights in the photo-consistency loss.\n2. A separate 3D and 4D hash feature grids to save the storage.  \n3. The optimization of mutual information between uncertainty and mask is demonstrated to be beneficial to the rendering quality.", "weaknesses": "1. The concept of representing scenes with separate static and dynamic parts has been investigated in NeRFPlayer or NeRF in the wild.  Integrating Bayesian learning to predict the dynamic points is also not new. \n2. Lack quantitative ablation study on the mutual information optimization, which weaken the contribution of this step. \n3. Inadequate references. There are missing references related to neural implicit representation for large scale and dynamic scenes:\n Xiuchao Wu et al., Scalable Neural Indoor Scene Rendering, SIGGRAPH 2022\n Haithem Turki et al., Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs, CVPR 2022\n Liao Wagn et al., Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time, CVPR 2022 Oral", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Weaknesses:\n1. >1.1.The concept of representing scenes with separate static and dynamic parts has been investigated in NeRFPlayer or NeRF in the wild.\n\n    Although the abstract concept of separating static and dynamic is not novel, many existing methods like NeRFPlayer, NeRFW, and MixVoxels adopt this strategy. We emphasize that our separation differs from previous approaches both in purpose (compact storage) and means (estimated by uncertainty). Our separation is soft which could help use very small hash tables to represent large numbers of spatiotemporal voxels without disastrous hash collisions, which improve the memory and rendering efficiency. In contrast, NeRFPlayer used three branches to ensemble the results for improving quality, while their design can not improve either memory or rendering efficiency. We summarize the differences in the following Table.\n    \n|                 |    NeRFPlayer     |    NeRF-W     |           MSTH(Ours)           |\n|:---------------:|:-----------------:|:-------------:|:------------------------------:|\n|     Purpose     |  improve quality  | remove noises | compactness (small hash table) |\n| How to separate | learnable network |  uncertainty  | uncertainty + MI maximization  |\n\n\n> 1.2 Integrating Bayesian learning to predict the dynamic points is also not new.\n\nWe indeed take inspiration from the successful work of NeRF-W. However,\nNeRF-W and our method aim to solve different problems, NeRF-W use uncertainty to reduce the impact of transient objects (noise), while they do not consider the reconstruction of transient objects. Instead, we treat the dynamic point as noise and they are also required to be reconstructed. With this important difference, we are required to make the uncertainty represent the dynamics through mutual information maximization. \n\n> 2. Lack quantitative ablation study on the mutual information optimization, which weaken the contribution of this step.\n\nIn Table 1 in the appendix, we ablate the Mutual information optimization by substituting the mutual information term with other descriptors of correlation including the Pearson correlation coefficient and a hard-coded linear correlation. Results demonstrate the effectiveness of the proposed Mutual information correlation. For clarity, we paste the comparison in the following table.\n\n\n\n| Correlation | PSNR$\\uparrow$ | LPIPS$\\downarrow$ |\n|:-----------:|:--------------:|:-----------------:|\n|    Mask     |     29.64      |       0.087       |\n|   Pearson   |     29.87      |       0.107       |\n| Hard-coded  |     28.74      |       0.124       |\n|   **MI**    |   **29.93**    |     **0.063**     |\n\n\n> 3. Inadequate references. There are missing references related to neural implicit representation for large scale and dynamic scenes: Xiuchao Wu et al., Scalable Neural Indoor Scene Rendering, SIGGRAPH 2022 Haithem Turki et al., Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs, CVPR 2022 Liao Wagn et al., Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time, CVPR 2022 Oral\n\nFor the missing references, they are indeed related and we will include them in the related work. (Fourier PlenOctrees [85] is already included in our draft)"}, {"review_id": "6FJR6uWZYh", "reviewer": "Reviewer_hvP1", "summary": "This paper present a method for efficient dynamic scene reconstruction. They represent the dynamic scene with a weighted combination of a 3D hash encoding (for static part) and a 4D hash encoding (for dynamic region). The weight is learnable and can be represented by a multi-resolution hash table or a 3D voxel grid. For each query 3D point, they can then get a weighted feature encoding from the hash tables. The photometric loss is then used to supervise the training. \n\nTo better supervise the learnable mask/weight representation, they further propose an uncertainty guided mask learning and exploit the mutual correlation between \"3D point uncertainty\" and \"3D point mask\" to supervise the learning of the mask representation. \n\nThe experimental evaluations are conducted on two public datasets and a self-collected dataset. The experimental results demonstrate the effectiveness of the proposed method. ", "questions": "The mask representation is not time dependent and the uncertainty supervision is based on the inconsistency between the rendered images & the real captured images. I have following two questions:\n\n1: Since the 3D scene is dynamic, the mask(static/dynamic) for each 3D position should also depend on time. Why you choose to use a time-independent representation. \n\n2: Is it like that as long as the 3D point being occupied by a dynamic object (even for partial frames), it will also be predicted as dynamic (which would have a high uncertainty from Eq. 7)? If so, will your method be able to work well if the dynamic object occupy a larger portion of the image and sweep over most of the 3D space across time. To what extent it can perform well?", "limitations": "If Question 2 is right, please address the corresponding limitation in Section 5 of the main paper. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1: The paper presents a novel method to improve both the reconstruction quality and efficiency for dynamic scenes. The experimentral results demonstrate its effectiveness. \n\n2: The paper is well written. ", "weaknesses": "I do not find severe flaws of the paper and would like to discuss with other reviewers if necessary. ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Since the 3D scene is dynamic, the mask(static/dynamic) for each 3D position should also depend on time. Why you choose to use a time-independent representation.\n\nWe design the time-independent mask mainly due to the inference efficiency. Specifically, we find the efficiency bottleneck of the hash encoder is mainly from the hash mapping (instead of the mlp due to the fast tiny-cuda-nn implementation). And for the hash mapping, the bottleneck is the memory access time. For a 3D mask, the hash mapping requires to access eight times of memory for a trilinear interpolation. If we use a 4D time-dependent mask, the hash mapping requires to access 16 times of memory for a 4-linear interpolation, which makes the inference time much slower than 3D masks. Besides, due to the extra time dimension, a 4D mask is also memory-inefficient while the performances we evaluated are similar with a 3D mask.\n\nConcretely, we also conduct experiments to compare the time-dependent and time-independent versions of masks. We compare them in the following table (on the Plenoptic Video dataset).\n\n| Metrics | PSNR | LPIPS | FPS | Memory |\n|:-------:|:----:|:-----:|:---:|:------:|\n| 3D Mask | 33.1 | 0.051 | 15  |  135M  |\n| 4D Mask | 32.7 | 0.053 |  9  |  183M  |\n\n\n> Q2: Is it like that as long as the 3D point being occupied by a dynamic object (even for partial frames), it will also be predicted as dynamic (which would have a high uncertainty from Eq. 7)? If so, will your method be able to work well if the dynamic object occupy a larger portion of the image and sweep over most of the 3D space across time. To what extent it can perform well?\n\nIn our implementation, if the dynamics of the underlying scene are of great complexity, our solution is to allocate more space for dynamic part by using a larger hash table. The main problem of the 4D mask is the inference efficiency which is mentioned in Q1. \n\nWe collect some complex multi-view videos in our proposed Campus dataset which exhibits long and large dynamic areas in our supplemental video. The proposed MSTH demonstrates commendable performance in the aforementioned scenes, despite the utilization of a limited-size dynamic hash table. This observation substantiates the algorithm's efficacy in handling scenes that encompass significant dynamic areas.\n\nFor most routine scenes, the 3D mask is enough for effectiveness and efficiency. We also believe the 4D time-dependent mask will be more memory efficient for extremely complex scenes, e.g., the dynamic areas are very large. We believe the more fine-grained 4D masks are a better solution in this situation for memory efficiency. We will also explore this situation more and sincerely thank the reviewer for giving this valuable feedback. "}, {"review_id": "YoBvAiBeK5", "reviewer": "Reviewer_Vvsv", "summary": "The paper presents MSTH, a method that efficiently reconstructs dynamic 3D scenes from multi-view or monocular videos. The proposed solution uses a space-time hash encoding, a prediction of masks, and uncertainty values that help a method to identify 3D points that belong to moving objects. The intuition is that this masking and uncertainty will help a method identify static points which are easy to reconstruct but also handle well the moving points. Lastly, the formulation of the method allows it to render in fast manner also preserving the size of the model. The paper presents experiments on three real-datasets: Plenoptic Video Dataset, Google Immersive Dataset, and a newly introduced dataset. The presented experiments show that the MSTH improves the PSNR while keeping both the training time and inference time short as well as maintaining a constant size model.", "questions": "Please see Weaknesses section.", "limitations": "I think the paper clearly states the limitations of the approach.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "In sum, I find the paper to be well executed and explained. Overall, I think the problem of reconstructing dynamic scenes is a challenging but important problem to solve. In detail here are the strengths I find:\n\nS1. I find the use of masking to identify the static and dynamic points an interesting and simple idea. I think the idea has good intuitive rationale and it should be effective.\n\nS2. The inclusion of uncertainty in the formulation I think is an important example of why uncertainty is very important in machine learning. I support the idea of combining uncertainty with the mask prediction as a mean to improve results.\n\nS3. The experiments use real-data and challenging scenes to showcase the benefits of the proposed approach. Overall, I think the experiments are well executed and highlight the benefits of the MSTH.\n\nS4. The clarity and description of MSTH is good. Overall, I think the proposed method should be easy to reproduced given the level of clarity of the narrative.", "weaknesses": "Overall, I find the paper to be well executed and clear. I only have one weakness to point out:\n\nW1. I think the paper is lacking a more in-depth discussion about the newly created dataset to test MSTH. Whil I understand that the algorithm is important, I also think the data part is as important as the algorithm. At the end of the day, the data is required and crucial to solve many problems in the era of deep learning. Thus, I suggest that a final version of the paper includes an extended discussion of the dataset curated by the authors.\n\nWhile I like the approach and explanation of MSTH, I do have some minor concerns:\n\nW2. The uncertainty described in the method in a way breaks with the intuitive understanding that an uncertainty value corresponds to a confidence value ranging from 0-1 and could be interpreted as a probability of correctness. However, the \"uncertainty\" introduced in Section 3.3 deviates from this common understanding. I think the paper should explicitly state a concrete meaning of uncertainty to clarify that the common understanding of uncertainty does not apply to this narrative. The main reason I bring this up is because I think this can confuse readers in the future.\n\nW3. In several parts of the narrative (e.g., lines 188 and 186) the paper uses the term \"dynamics of a point\". I find this term to be confusing since in many fields (e.g., robotics or control) the dynamics typically refer to a model that describes how the state of something changes over time. I think the paper should clarify what this exactly means to clear any misconception of this term.\n\nW4. Missing definition of different terms: 1) $\\tilde{m}(\\cdot)$ is never defined in the text; 2) what does \"large $m$\" means in line 168? Isn't $m$ the learned mask? Or is it a value?\n\n----\nPost Rebuttal\n\nAfter the rebuttal, all my concerns were clarified and I still think this is a good contribution. Thus, I maintain my rating.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> W1: I think the paper is lacking a more in-depth discussion about the newly created dataset to test MSTH. While I understand that the algorithm is important, I also think the data part is as important as the algorithm. At the end of the day, the data is required and crucial to solve many problems in the era of deep learning. Thus, I suggest that a final version of the paper includes an extended discussion of the dataset curated by the authors.\n\n\nDue to the line limit for the submission, we only provide a concise description of the proposed Campus dataset in the main text. A more detailed introduction and configuration can be found in the Appendix. We would provide a more comprehensive explication of the dataset within the main body.\n\n\n> W2: The uncertainty described in the method in a way breaks with the intuitive understanding that an uncertainty value corresponds to a confidence value ranging from 0-1 and could be interpreted as a probability of correctness. However, the \"uncertainty\" introduced in Section 3.3 deviates from this common understanding. I think the paper should explicitly state a concrete meaning of uncertainty to clarify that the common understanding of uncertainty does not apply to this narrative. The main reason I bring this up is because I think this can confuse readers in the future.\n\nThe uncertainty we used in this paper is derived from the *Aleatoric Uncertainty* proposed in [1] which model the intrinsic noise of the observed data by Gaussian distributions. The uncertainty $U(\\cdot)$ stands for the learned standard deviation of the underlying Gaussian which is unbounded by its definition. The uncertainty augmented loss would assign a relatively large uncertainty value on those spatial points with high time variance which helps the model in the separation of static and dynamic regions.\n\n[1] Kendall, Alex, and Yarin Gal. \"What uncertainties do we need in Bayesian deep learning for computer vision?.\" Advances in neural information processing systems 30 (2017).\n\n\n> W3: In several parts of the narrative (e.g., lines 188 and 186) the paper uses the term \"dynamics of a point\". I find this term to be confusing since in many fields (e.g., robotics or control) the dynamics typically refer to a model that describes how the state of something changes over time. I think the paper should clarify what this exactly means to clear any misconception of this term.\n\n\nWe thank the reviewer for pointing out the confusion arising from the lack of clarity in our submission. In the context of our paper, \"dynamics of a point\" \nto how frequently the state of density and color changes through time, e.g, a more dynamic point will have more density and color states through time, which can be reflected by the uncertainty. In contrast, the traditional understanding of dynamics within robotics and control refers to a model explicitly formulated and constructed by human intervention, which is devised to capture and predict the behavior of an object. In response to the concerns raised regarding potentially misleading representations, we assert our commitment to rectify ambiguities and refine the clarity of terminology in the refined version of our paper.\n\n> W4: Missing definition of different terms: 1) $\\tilde{m}(\\cdot)$ is never defined in the text; 2) what does $m$ means in line 168? Isn't the learned mask? Or is it a value?\n\nWe thank the reviewer for pointing out the potential missing definitions. The symbol $\\tilde{m}$ in Eq(2) refers to the unactivated mask value, namely the raw output of the mask branch of the neural network. This $\\tilde{m}(\\cdot)$ is converted to the final mask value by $\\text{sigmoid}$ function, as shown in Eq(2). \nThe $m$ in line 168 stands for the same meaning as that in other equations, namely the corresponding mask value. Our symbol system is consistent in the paper, the $m$ is firstly defined in Eq(2) and retains the same meaning in line 168. "}, {"review_id": "CSTetTrHgX", "reviewer": "Reviewer_ohyg", "summary": "This paper proposes the masked space-time hash encoding to efficiently reconstruct dynamic scenes. The insights behind the paper is that: most part of the scene is static, simply modelling such static parts can dramatically increase the probability of hash collisions; while increasing the hash table entries requires more memory. To solve these issues, this paper proposes to decouple the scene into a static part and a dynamic part, where the two parts are jointly learned through an uncertainty mask. The uncertainty mask is modeled by an individual voxel grid, on which each voxel stores an value of the uncertainty field. To bridge the gap between uncertainties and masks, a neural estimator is adopted to approximate the mutual information. Experiments are conducted on the Plenoptic Video Dataset, the Google Immersive Dataset, and the self-collected time-synchronized multi-view videos. As shown in the quantitative results, MSTH surpasses state-of-the-art dynamic NeRF methods (HexPlane, K-Planes, etc) in terms of both the reconstruction quality and training/rendering time. Ablation studies also shows the effectiveness of the proposed masked space-time hash encoding.", "questions": "- (1)  Is Figure 3(c) revealing that the uncertainty does not accurately model the static part and dynamic part?\n- (2) In Section 4.3, the authors proposed two variants for ablation study: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Actually, for the second variant, I'm not quite sure whether the authors split the scene into a static part or a dynamic part using additional models (for example, using the **segment anything model (SAM)** to split a non-rigid object/human from the static background), and then separately model them. If it is not, I wonder would it better to separately model the dynamic part and static part with some pre-computed masks instead of jointly learn the uncertainty during training?", "limitations": "The authors address the major limitations of existing dynamic NeRF methods, e.g. the efficiency and rendering quality. Some other limitations, such as reconstructing monocular dynamic scenes, blurry scenes, etc, are mentioned in the paper.", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "This is a technically very solid paper. Compared to previous methods, the proposed method can reconstruct dynamic scenes with higher quality while highly reduced the training time. I like the idea of using uncertainty to mask the static part and dynamic part in a scene instead of simply a combination of 3D voxel grid and a 4D voxel grid. The experiments are exhaustive and highly support the effectiveness of the proposed method. ", "weaknesses": "(1) In Figure 3 (c), it seems some static parts have high uncertainty values (the top-left part and the middle-right part)?\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> Q1: Is Figure 3 \\(c\\) revealing that the uncertainty does not accurately model the static part and dynamic part?\n\n\nThe dynamic region inferred by the model may include some noise induced by algorithm-irrelevant reasons, such as inaccurate estimated camera poses and parameters, lack of key points when the scene contains little high-frequency information and etc. In our experiment, the upper-left part of the scene could not be well reconstructed even by exploiting per-frame static nerf algorithms like Instant-NGP. As a result, we consider this as a data intrinsic noise that has little impact on the visual quality.\n\n> Q2: In Section 4.3, the authors proposed two variants for ablation study: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Actually, for the second variant, I'm not quite sure whether the authors split the scene into a static part or a dynamic part using additional models (for example, using the segment anything model (SAM) to split a non-rigid object/human from the static background), and then separately model them. If it is not, I wonder would it better to separately model the dynamic part and static part with some pre-computed masks instead of jointly learn the uncertainty during training?\n\nFor the variant \"A simple decomposition which is an addition of a 3D hash table and a 4D hash table\" in the ablation study part, we did not apply any off-the-shelf segmentation model to split the dynamic part from the static part. In this ablation study, the separation of the static and dynamic part is learned implicitly through reconstruction loss without the guidance of uncertainty, which results in sub-optimal performance as shown in Fig 6 in the paper, and proves the effectiveness of the proposed uncertainty estimation. \n\nIncorporating segmentation models in the separation of static and dynamic regions introduces a strong prior that the dynamic region consists of semantically-grounded objects. Besides, the identification of dynamic objects requires manual specification. This assumption is not reasonable in many situations. For example, in the *flame salmon* scene in the Plenoptic dataset, some foreground objects (including the cups, the table and etc.) are actually static. Apart from that, some dynamic regions may not manifest within the vocabulary of a particular segmentation algorithm. Even if the dynamic part can be roughly captured, the concrete bounds cannot be well inferred without other guidance (e.g. the hands and head of the cooking man in the *flame salmon* scene), and tracking the dynamic objects in multiple videos also introduces extra complexity. \n\nIn contrast, our solution provides a more general framework that requires no prior on the dynamic scenes, in which the granularity of dynamic region is automatically determined under the guidance of uncertainty. "}]

Author rebuttal:

Meta-review: The final score of this paper is A, A, WA, BA. All reviewers agree that this paper has made contributions and the proposed method is effective in speeding up the training and reducing the memory storage required for dynamic scene reconstruction. The rebuttal has done an excellent job in providing additional experiments and explanations. Please incorporate all the additional materials in rebuttal into the final version of the paper.
Decision: Accept (spotlight)