Title: Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: NeRF

Dynamic Scenes
Abstract: In this paper, we propose the Masked Space-Time Hash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size.Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene. We evaluate our method on extensive dynamic scenes. As a result, MSTH obtains consistently better results than previous state-of-the-art methods with only 20 minutes of training time and 130 MB of memory storage.

Masked Space-Time Hash Encoding for Efficient Dynamic Scene Reconstruction

Abstract

In this paper, we propose the M asked S paceT ime H ash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size. Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene. As a result, MSTH obtains consistently better results than previous methods with only 20 minutes of training time and 130 MB of memory storage. Code is available at https://github.com/masked-spacetime-hashing/msth .

1 Introduction

Neural radiance fields [57, 96, 79, 58, 14] have achieved great success in reconstructing static 3D scenes. The learned volumetric representations can produce photo-realistic rendering for novel views. However, for dynamic scenes, the advancements lag behind that of static scenes due to the inherent complexities caused by the additional time dimension. Specifically, reconstructing dynamic scenes requires more time, memory footprint, and additional temporal consistency compared to static scenes. These factors have resulted in unsatisfactory rendering qualities. Recently, many works [40, 38, 75, 83, 1, 10, 72] have made great progress in dynamic scene reconstruction, im-

0.3

1

2

3

6

9

12

Training time (hour)

30

31

32

33

PSNR on Plenoptic Video

Ours

Ours

†

MixVoxels [83]

K-Planes [72]

NeRFPlayer [75]

HexPlane [10]

HyperReel [1]

HexPlane

†

[10]

27

28

29

30

PSNR on Google Immersive

Ours

NeRFPlayer [75]

HyperReel [1]

Plenoptic

Dataset

Immersive Dataset

proving the efficiency and effectiveness of reconstructions. However, there is still considerable room

DyNeRF[40] 1344h

HexPlane[10] 12h

Ours 20min

NeRFPlayer[75] 6h

HyperReel[1] 2.7h

Ours 20min

DyNeRF[40] 1344h

HyperReel[1] 2.7h

Ours 20min

for improvement in many aspects such as rendering quality, motion coherence, training and rendering speed, and memory usage.

This paper aims to develop a method for reconstructing dynamic scenes with high rendering quality in a space- and time-efficient manner. To achieve this goal, we base our approach on the multi-resolution hash encoding [58] due to its efficiency and compactness for representing and reconstructing static scenes. However, directly applying a 4D multi-resolution hash table to represent a dynamic scene would require a much larger hash table size than that for a static scene due to the much more hash collisions caused by the additional time dimension. Our finding is that, for the ubiquitous existence of static areas in a scene, storing the static points into a 4D time-dependent hash table will result in an information redundancy since each of them will occupy T hash table items with an identical value for a T -frame scene. It will also lead to a high hash collision rate since it narrows the capacity of the hash table, which negatively impacts the reconstruction quality. Therefore, for the points with low dynamics, we hope to establish a mechanism to reduce the frequent queries and updates to the 4D hash table and automatically save their features into a 3D hash table to avoid temporal redundancy.

From this observation, we propose Masked Space-Time Hash encoding, which combines a 3D hash encoder and a 4D hash encoder with a 3D learnable weight mask. In order to make the mask correlate with the dynamics of the corresponding positions, we adopt the Bayesian framework of Kendall and Gal [34] to estimate the uncertainty [34, 54] of each point being static. The non-linear correlation [6] between the uncertainty and the learnable mask is maximized to make the mask reflect the dynamics. In this way, the static points indicated with a low uncertainty will have a low weight for the 4D hash table and a high weight for the 3D hash table, which prevents modifications to the dynamic representations. With the proposed masked space-time hash encoding, we can set the size of a 4D hash table the same as a 3D one without much loss of rendering qualities, making the representation highly compact. Besides, without the requirements to fit the large numbers of repeated features independently, our method is easier to optimize and converge rapidly in only twenty minutes. To validate the effectiveness of our method on scenes in more realistic settings with large areas of dynamic regions and more complex movements, we collect a synchronized multi-view video dataset with 6 challenging dynamic scenes, which will be publicly available. As a result, the proposed masked space-time hash encoding achieves consistently better reconstruction metrics on two publicly available datasets consisting of 13 scenes, with only 20 minutes of training and 130 MB of storage. Fig. 1 and Fig. 2 show the quantitative and qualitative comparisons to other state-of-the-art methods. In summary, our contributions are:

We propose Masked Space-time Hash Encoding, a novel representation that decomposes the dynamic radiance fields into a weighted combination of 3D and 4D hash encoders.

The proposed method is validated on a wide range of dynamic scenes, surpassing previous state-of-the-art methods by non-trivial margins with only 20 minutes of training time and 130 MB of memory storage.

We propose a new synchronized multi-view dataset with more challenging dynamic scenes, including scenes with many moving objects and scenes with complex and rapid movements.

2 Related Work

Neural Scene Representations for Static Scenes. Representing 3D scenes with neural networks has achieved remarkable success in recent years. NeRF [57] first proposes to use neural networks to represent radiance fields, demonstrating the remarkable potential of this representation in conjunction with volume rendering techniques. The high-fidelity rendering has sparked a surge of interest in related areas. Numerous variants have emerged to address the inherent challenges of the original approach, including improving rendering quality [3, 5, 35, 52, 90], handling sparse input [60, 13, 92, 55, 94, 17], lighting [8, 76, 62], editing [37, 48, 45, 20, 56], dealing with complicated [86, 25, 95, 36] and large scenes [81, 100, 54, 15], generalization [98, 73, 80, 12, 87, 59, 29, 88], jointly optimizing with camera poses [43, 89, 7, 30, 78, 103, 16], accelerating training [96, 14, 79, 93, 18] and speeding up rendering [71, 26, 97, 70, 47, 69, 44, 24, 28, 11, 84, 51]. Our work draws inspiration from two recent contributions in the field, namely Instant-NGP [58] and Mip-NeRF 360 [4]. In InstantNGP, Müller et al. [58] proposes a novel data structure that leverages multi-resolution hash grids to efficiently encode the underlying voxel grid while addressing hash collisions using a decoding MLP. Furthermore, the proposed method allows for fast and memory-efficient rendering of large-scale scenes. Similarly, Mip-NeRF 360 [4] presents a sample-efficient scheme for unbound scenes, which utilizes a small density field as a sample generator and parameterizes the unbounded scene with spherical contraction. In this paper, we build upon these contributions and propose a novel approach that extends successful techniques and components to incorporate time dimension with minimal overhead. Our method is capable of representing a dynamic scene with only 2~3 × memory footprint than that of a static NeRF.

Novel View Synthesis for Dynamic Scenes. Extending the neural radiance field to express dynamic scenes is a natural yet challenging task that has been proven crucial to many downstream applications [39, 66, 77, 31]. Many researchers focus on monocular dynamic novel view synthesis, which takes a monocular video as input and targets to reconstruct the underlying 4D information by modeling deformation explicitly [19, 22, 41, 91, 68, 99] or implicitly [64, 23, 63, 21, 74, 49, 46, 102, 82, 32]. As an exemplary work, D-NeRF [68] models a time-varying field through a deformation network that maps a 4D coordinate into a spatial point in canonical space. Despite the great outcomes achieved by research in this line, the applicability of these methods is restricted by the inherent nature of the underlying problem [23]. A more practical way of reconstructing dynamic scenes is by employing multi-view synchronized videos [33, 50, 104, 38, 1, 75, 10, 83, 85, 2, 67]. DyNeRF [40] models dynamic scenes by exploiting a 6D plenoptic MLP with time queries and a set of differencebased importance sampling strategies. The authors also contributed to the field by presenting a real-world dataset, which validated their proposed methodology and provided a valuable resource for future research endeavors. K-Planes [72] and HexPlane [10] speed up training by decomposing the underlying 4D radiance field into several low-dimensional planes, which substantially reduces the required memory footprint and computational complexity compared with explicit 4D voxel grid. HyperReel [1] breaks down the videos into keyframe-based segments and predicts spatial offset towards the nearest keyframe. NeRFPlayer [75] and MixVoxel [83] address the problem by decomposing 4D space according to corresponding temporal properties. The former separates the original space into three kinds of fields and applies different structures and training strategies, while the latter decouples static and dynamic voxels with a variational field which further facilitates high-quality reconstruction and fast rendering. Our method implicitly decomposes 3D space with a learnable mask. This mask eliminates the requirement for manual determination of dynamic pixels and enables the acquisition of uncertainty information, facilitating high-quality reconstruction.

3 Methodology

Our objective is to reconstruct dynamic scenes from a collection of multi-view or monocular videos with a compact representation while achieving fast training and rendering speeds. To this end, we propose Masked Space-Time Hash encoding, which represents a dynamic radiance field by a weighted combination of a 3D and a 4D hash encoder. We employ an uncertainty-guided mask learning strategy to enable the mask to learn the dynamics. In the following, we will first introduce the preliminary, then the masked space-time hash encoding and uncertainty-guided mask learning, and finally the ray sampling method.

3.1 Preliminary

For neural radiance fields, the input is a 3D space coordinate x and a direction d to enable the radiance field to represent the non-Lambertian effects. The output is the color c ( x , d ) ∈ R 3 and a direction-irrelevant density σ ( x ) ∈ R . Most existing methods encode the input coordinate with a mapping function h that maps the raw coordinate into Fourier features [57], voxels-based features [79, 96], hashing-based features [58] or factorized low-rank features [14]. In this work, we mainly focus on the multi-resolution hash encoding due to its efficiency and compactness. Specifically, for an input point x , the corresponding voxel index is computed through the scale of each level, and a hash table index is computed by a bit-wise XOR operation [61]. There are L levels of hash tables corresponding to different grid sizes in a geometric progressive manner. For each level, the feature for a continuous point is tri-linearly interpolated by the nearest grid points. The corresponding features in different levels are concatenated to obtain the final encoding. For convenience, we denote the output of the multi-resolution hash encoder for x as enc ( x ) .

After the multi-resolution hash encoder, a density-decoding MLP ϕ θ 1 is employed to obtain the density and an intermediate feature g such that: σ ( x ) , g ( x ) = ϕ θ 1 ( enc ( x )) . Then, the color is computed through a color-decoding MLP ψ θ 2 : c ( x ) = ψ θ 2 ( g ( x ) , d ) which is direction-dependent.

For rendering, points along a specific ray r ( s ) = o + s d are sampled and the volumetric rendering [57] is applied to get the rendered color ˆ C ( r ) :

A squared error between the rendered color ˆ C ( r ) and the ground truth color C ( r ) is applied for back-propagation.

3.2 Masked Space-Time Hashing

For a dynamic neural radiance field, the input is a 4D space-time coordinate ( x , t ) . A straightforward method is to replace the 3D hash table with a 4D one. However, this simple replacement will result in a high hash collision rate due to the enormous volume of hash queries and modifications brought by the additional time dimension. For instance, in a dynamic scene comprising T frames, the hash collision rate is T times higher than a static scene, leading to a degradation in the reconstruction performance. Enlarging the size of the hash table will cause an unbearable model size and be difficult to scale up with the frame numbers.

To solve this problem, we propose the masked space-time hash encoding, which incorporates a 3D multi-resolution hash mapping h 3D , a 4D multi-resolution hash mapping h 4D , and a learnable mask encoding ˜ m . The final encoding function is formulated as follows:

The learnable mask m can be represented by a multi-resolution hash table or a 3D voxel grid. For the 4D hash table, the sizes of time steps also adopt a geometric growing multi-resolution scheme, which is reasonable due to the natural hierarchical properties of motions in time scales.

After obtaining the space-time encoding, the density-decoding and color-decoding MLPs are applied to obtain the final outputs:

Then the volumetric rendering is applied to each ray at each time step, and a squared error is employed as the loss function:

Reduce Hash Collisions. The intuition behind the masked modeling lies in the fact that many parts in the dynamic radiance fields are time-invariant, such as the static objects, the background, etc. These static parts can be well reconstructed from the 3D hash table with a large m . For these static points, storing their features in the 4D hash table will occupy a large number of storage, largely increasing the hash collision rate. For these time-invariant parts, the 3D hash encoder can be sufficient to reconstruct, and these static parts will not modify the 4D hash table significantly when (1 -m ( x )) is small. In this way, the 4D hash table stores the properties of those points which are really dynamic, and those dynamic features in the 4D hash table will be protected by the mask term to suppress the gradients from static points.

Accelerate Rendering. Another advantage of using masked space-time hash encoding is to accelerate the fixed view-port rendering. Instead of rendering a novel view video frame-by-frame, we adopt an incremental way of using the mask to avoid redundant computations. Specifically, for a T -frame scene, we first render the initial frame V 0 as usual. The obtained mask is used to filter the static parts, and we only render the dynamic parts of other frames with the dynamic weight (1 -m ( x )) > (1 -ϵ ) , where ϵ is a hyper-parameter. In this way, except for the initial frame, we only require to render the dynamic part, improving the rendering fps from 1 . 4 to 15 , without loss of rendering quality.

The key to the masked space-time hash encoding is that m ( x ) can reflect the dynamics of the point. To this end, we design an uncertainty-guided mask learning strategy to connect the relation between the 3D hash uncertainty u ( x ) and the mask m ( x ) . We will elaborate on this strategy in the next part.

3.3 Uncertainty-guided Mask Learning

To make the mask m ( x ) well reflect the dynamics of the corresponding point, we design an uncertainty branch in our model to estimate the uncertainty of a point being static, which is a good indicator for the dynamics of the point. To this end, the uncertainty branch is required to reconstruct the dynamic scenes only using the 3D hash table with the input-dependent uncertainties, ignoring the time input. In this way, the dynamic points are regarded as the inherent noise since its supervision are inherently inconsistent (different time step describes different geometries while the uncertainty model is time-agnostic). The inherent noise of dynamic points will lead to a high uncertainty. We adopt the Bayesian learning framework of Kendall and Gal [34] to model the heteroscedastic aleatoric uncertainty for each point.

Specifically, we construct an uncertainty field u with a voxel grid representation. u ( x ) denotes the uncertainty of a space point x . We denote the raw output of the uncertainty voxel-grid as ˜ u , and a soft-plus is used as the activation: u ( x ) = u m +log (1 + exp(˜ u ( x ))) , where u m is a hyper-parameter for shifting the uncertainty values [54]. For each ray, the ray-level uncertainty U( r ) is calculated through volumetric rendering [53]:

Besides, the color and density estimated by the 3D hash table are:

The rendered color of this branch is ˆ C s ( r ) by applying the volumetric rendering. After that, the uncertainty-based loss for ray r is defined as:

Note that the uncertainty branch is time-agnostic , i.e., the predicted color ˆ C ( r ) is not time-dependent, which is important to make the uncertainty relevant to dynamics. In this way, the uncertainty for each point u ( x ) can be estimated through the above loss function. For the dynamic points, the uncertainty is inherently large because of the ambiguous and inconsistent geometries caused by the inconsistent pixel color supervision. In this perspective, the uncertainty can well reflect the dynamics of each point and could guide the mask values.

Bridging uncertainty with mask. Though we find the correlation between the mask m and uncertainty u , it is not trivial to connect them. First, m and u are in different value ranges, m ( x ) ∈ [0 , 1] while u ( x ) ∈ [0 , + ∞ ) . Second, the distributions of m and u are very different, and the relations between them are non-linear. Imposing a hard relationship between them will impact the training of their specific branches. We instead maximize the mutual information between the two random variables m and u to maximize the nonlinear correlation between them. Although MI can not measure whether the correlation is negative, Eq. (7) can guarantee this. The mutual information I ( m,u ) describes the decrease of uncertainty in m given u : I ( m,u ) := H ( m ) -H ( m | u ) , where H is the Shannon entropy. To estimate the mutual information between m and u , we adopt the neural estimator in [6] to approximate the mutual information I ( m,u ) as:

P m,u is the joint probability distribution of m and u , and P m ⊗ P u is the product of marginals. T θ is a neural network with parameters θ ∈ Θ . We choose a small MLP with two hidden layers to represent T θ and draw samples from the joint distribution and the product marginals to compute the empirical estimation of the expectation. By maximizing the estimated mutual information, we build the correlation between m and u . At last, the overall learning objective of MSTH is the combination of the above losses:

where λ and γ are two hyper-parameters. For rendering a novel view, the uncertainty branches and the mutual information network are disabled.

In practice, the model can learn a reasonable mask m even without the uncertainty guidance. However, this will make the learned mask very noisy and tend to learn a "middle value" in [0 , 1] , which will make the static points still have a relatively large dynamic weight to modify the 4D table. Fig. 3 visualize the learned mask with or without uncertainty guidance. With uncertainty guidance, the mask will tend to be binarized and the static parts are with very low dynamic weight. For the mutual information constraint, we find it will make the distribution of m towards a Bernoulli distribution which is helpful for reducing hash collision and accelerating rendering speed. Without the constraint, the model tends to learn more from the 3D hash table only even for the dynamic point. This will make the failing captures of some dynamic voxels with transient changes. As a result, the uncertainty guidance will help learn a finer detail, which will be shown in the ablation part.

3.4 Ray Sampling

For a natural video, a significant portion of the scene is usually static or exhibits only minor changes in radiance at a specific time across the entire video. Uniformly sampling the space-time rays causes an imbalance between the static and dynamic observations. Therefore, we sample the space-time queries according to the quantification of dynamics. Specifically, we sample a space-time ray ( r , t ) according to a pre-computed probability P ( r , t ) . We decompose the sampling process into two sub-process: spatial sampling and temporal sampling. Formally, we decompose P ( r , t ) as P ( r , t ) = P ( t | r ) P ( r ) , which forms a simple Markov process that first sample the space ray according to P ( r ) then sample the time step by P ( t | r ) .

Plenoptic Video Dataset

Immersive Dataset

Campus Dataset

In practice, we build P ( r ) as the softmax of temporal standard deviation std ( r ) of the corresponding pixel intensities { C ( r, t ) } t :

where τ 1 is the temperature parameter. We set a higher temperature for a softer probability distribution [27]. For temporal sampling, we follow [42] to compute the weight according to the residual difference of its color to the global median value across time ¯ C ( r ) with temperature τ 2 :

3.5 Implementation Details

To improve the sampling efficiency, we utilize the proposal sampler [4] that models density at a coarse granularity with a much smaller field, thereby generating more accurate samples that align with the actual density distribution with minor overhead. In our implementation, we use one layer of proposal net to sample 128 points. For the mask, we use a non-hash voxel grid with 128 spatial resolution. To encourage the separation of the static and the dynamic, we utilize a mask loss that aims at generating sparse dynamic voxels by constraining the mask to be close at 1. We also adopt distortion loss [4] with λ dist = 2 e -2 . For uncertainty loss, we set γ = 3 e -4 and λ = 3 e -5 . We find a small coefficient of mutual information estimator γ will have a large impact on the gradients, which is consistent with the observations of [6]. For the hash grid, we implement a CUDA extension based on PyTorch [65] to support the rectangular voxel grid required by the proposed method. For the complete experiment setting and model hyper-parameters, please refer to the Appendix.

4 Experiments

4.1 Dataset

For validating the performances of the proposed method, we conduct experiments on two public datasets and our collected dataset: (1) The Plenoptic Video Dataset [40], which consists of 6 publicly accessible scenes: coffee-martini, flame-salmon, cook-spinach, cut-roasted-beef, flame-steak, and sear-steak. Each scene contains 19 videos with different camera views. The dataset contains some challenging scenes, including objects with topology changes, objects with volumetric effects, various lighting conditions, etc. (2) Google Immersive Dataset [9]: The Google Immersive dataset contains light field videos of different indoor and outdoor environments captured by a time-synchronized 46-fisheye camera array. We use the same 7 scenes ( Welder , Flames , Truck , Exhibit , Alexa Meade

Exihibit , Face Paint 1 , Face Paint 2) as NeRFPlayer [75] for evaluation. (3) To validate the robustness of our method on more complex in-the-wild scenarios, we collect six time-synchronized multi-view videos including more realistic observations such as pedestrians, moving cars, and grasses with people playing. We named the collected dataset as Campus Dataset . The Campus dataset is much more difficult than the above two curated ones in the movement complexities and dynamic areas. For detail on the dataset, please see our Appendix. For the above three multi-view datasets, we follow the experiment setting in [40] that employs 18 views for training and 1 view for evaluation. To quantitatively evaluate the rendering quality on novel views, we measure PSNR, DSSIM, and LPIPS [101] on the test views. We follow the setting of [40] to evaluate our model frame by frame. Our method is also applicable to monocular videos. To validate the reconstruction quality with monocular input, we conduct experiments on D-NeRF [68] dataset, which contains eight videos of varying duration, ranging from 50 frames to 200 frames per video. There is only one single training image for each time step. For evaluation, we follow the common setting from [68, 72, 10].

4.2 Results

Multi-view Dynamic Scenes. Wereconstruct the dynamic scenes from multi-view time-synchronized observations on the two public multi-view video datasets and our collected Campus dataset. The quantitative results and comparisons are presented in Tab. 1 and Tab. 2. For the Plenoptic Video dataset, our method surpasses previous state-of-the-art methods by a non-trivial margin, with a 0 . 7 to 1 . 4 PSNR gains, and > 30% improvements on the perceptual metric LPIPS. For training, we cost at most 1 / 5 training time of other methods, achieving a speedup of 5 -36 compared with other fast reconstruction algorithms and a 4000 speedup compared with DyNeRF [40]. Besides, we also keep a compact model size with only 135 MBof memory occupancy, showing the advantages of the masking strategy which can avoid a large number of hash collisions and make the dynamic hash table size smaller enough. For the Google Immersive dataset, Tab. 2 also shows consistently non-trivial improvements in both training efficiency and reconstruction quality.

Fig. 2 visualizes the qualitative comparisons of our method to other state-of-the-art methods, from which we can observe that our method can capture finer motion details than other methods. For example, MSTH can well reconstruct the fire gun with distinct boundaries and specular reflection, the mark of the hat, and the stripe of the salmon. The Splashing sparks can also be accurately captured. We provide the representative novel-view rendering results of our method in Fig. 4.

Monocular Dynamic Scenes. We present the quantitative and qualitative results of our proposed method on the D-NeRF dataset in Tab. 3 and Fig. 5, accompanied by a comparative analysis with related approaches. MSTH outperforms all other methods in terms of SSIM and LPIPS, including methods that focus on dynamic scenes (K-Planes [72] and HexPlane [10]) and TiNeuVox, the current state-of-the-art method for monocular dynamic novel view synthesis. The proposed MSTH achieves superior results without any assumption about the underlying field which demonstrates that our method could be easily extended to accommodate scenarios with varying complexity and dynamics.

外

4.3 Ablation Study

Ablation on the Masked Hash Encoding. To evaluate the effect of decomposing a 4D dynamic radiance field into the proposed masked space-time hash encoding, we propose two variants as comparisons: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Fig. 6 (a) and (b) visualize the qualitative comparisons. From Fig. 6 (a) we can observe that only using the 4D hash table will generate blurry rendering with a 3 PSNR drop. With a simple addition method, the reconstruction quality is improved compared with only a 4D hash table, while the dynamic regions are not captured well. Fig. 6 (b) and (c) show the comparisons of MSTH with the addition. MSTH improves the LPIPS by 20% compared with the addition method.

Ablation on the uncertainty-based guidance. We visualize the learned mask with or without uncertainty-based guidance, Fig. 3 shows the comparison. The learned mask with uncertainty loss will distinguish the static and dynamic parts more clearly, with less noisy judgment, leading to an assertive distribution on the mask, which is beneficial to avoid hash collisions. Fig. 6 (d) visualize the qualitative comparisons. MSTH with uncertainty loss performs better on the motion details.

5 Limitations and Conclusion

Limitations. MSTH tends to generate unsatisfying results when detailed dynamic information is insufficient, especially in monocular dynamic scenes, since it relies solely on the input images. When the input is blurred, occluded, or incomplete, the method may struggle to infer the motion information, leading to artifacts such as ghosting, flickering, or misalignment, which may degrade the visual quality. Further research is needed to develop advanced methods that handle complex scenes, motion dynamics and integrate multiple information sources to enhance synthesis accuracy and robustness.

Conclusion. In this paper, we propose a new method to reconstruct dynamic 3D scenes in a timeand space-efficient way. We decouple the representations of the dynamic radiance field into a timeinvariant 3D hash encoding and a time-variant 4D hash encoding. With an uncertainty-guided mask as weight, our algorithm can avoid a large number of hash collisions brought about by the additional time dimension. We conduct extensive experiments to validate our method and achieve state-ofthe-art performances with only 20 minutes of training. Besides, we collect a complex in-the-wild multi-view video dataset for evaluating the robustness of our approach on more realistic dynamic

(a)

(b)

(c)

(d)

scenes, including many daily activities. We hope our method can serve as a fast and lightweight baseline to reconstruct dynamic 3D scenes, which may derive many valuable applications such as interactively free-viewpoint control for movies, cinematic effects, novel view replays for sporting events, and other VR/AR applications.

References

B. Attal, J.-B. Huang, C. Richardt, M. Zollhoefer, J. Kopf, M. O'Toole, and C. Kim. Hyperreel: Highfidelity 6-dof video with ray-conditioned sampling. arXiv preprint arXiv:2301.02238 , 2023.

A. Bansal, M. Vo, Y. Sheikh, D. Ramanan, and S. Narasimhan. 4d visualization of dynamic events from unconstrained multi-view videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5366-5375, 2020.

J. T. Barron, B. Mildenhall, M. Tancik, P. Hedman, R. Martin-Brualla, and P. P. Srinivasan. Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5855-5864, 2021.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5470-5479, 2022.

J. T. Barron, B. Mildenhall, D. Verbin, P. P. Srinivasan, and P. Hedman. Zip-nerf: Anti-aliased grid-based neural radiance fields. arXiv , 2023.

M. I. Belghazi, A. Baratin, S. Rajeswar, S. Ozair, Y . Bengio, A. Courville, and R. D. Hjelm. Mine: mutual information neural estimation. arXiv preprint arXiv:1801.04062 , 2018.

W. Bian, Z. Wang, K. Li, J. Bian, and V. A. Prisacariu. Nope-nerf: Optimising neural radiance field with no pose prior. 2023.

M. Boss, R. Braun, V. Jampani, J. T. Barron, C. Liu, and H. Lensch. NeRD: Neural reflectance decomposition from image collections. https://arxiv.org/abs/2012.03918 , 2020.

M. Broxton, J. Flynn, R. Overbeck, D. Erickson, P. Hedman, M. Duvall, J. Dourgarian, J. Busch, M. Whalen, and P. Debevec. Immersive light field video with a layered mesh representation. ACM Transactions on Graphics (TOG) , 39(4):86-1, 2020.

A. Cao and J. Johnson. Hexplane: a fast representation for dynamic scenes. arXiv preprint arXiv:2301.09632 , 2023.

J. Cao, H. Wang, P. Chemerys, V. Shakhrai, J. Hu, Y. Fu, D. Makoviichuk, S. Tulyakov, and J. Ren. Real-time neural light field on mobile devices. arXiv preprint arXiv:2212.08057 , 2022.

E. Chan, M. Monteiro, P. Kellnhofer, J. Wu, and G. Wetzstein. pi-GAN: Periodic implicit generative adversarial networks for 3D-aware image synthesis. https://arxiv.org/abs/2012.00926 , 2020.

A. Chen, Z. Xu, F. Zhao, X. Zhang, F. Xiang, J. Yu, and H. Su. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14124-14133, 2021.

A. Chen, Z. Xu, A. Geiger, J. Yu, and H. Su. Tensorf: Tensorial radiance fields. arXiv preprint arXiv:2203.09517 , 2022.

X. Chen, Q. Zhang, X. Li, Y. Chen, F. Ying, X. Wang, and J. Wang. Hallucinated neural radiance fields in the wild, 2021.

Y. Chen, X. Chen, X. Wang, Q. Zhang, Y. Guo, Y. Shan, and F. Wang. Local-to-global registration for bundle-adjusting neural radiance fields. arXiv preprint arXiv:2211.11505 , 2022.

J. Chibane, A. Bansal, V. Lazova, and G. Pons-Moll. Stereo radiance fields (srf): Learning view synthesis from sparse views of novel scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE, jun 2021.

K. Deng, A. Liu, J.-Y. Zhu, , and D. Ramanan. Depth-supervised nerf: Fewer views and faster training for free. arXiv preprint arXiv:2107.02791 , 2021.

Y. Du, Y. Zhang, H.-X. Yu, J. B. Tenenbaum, and J. Wu. Neural radiance flow for 4d view synthesis and video processing. In Proceedings of the IEEE/CVF International Conference on Computer Vision , 2021.

Z. Fan, Y. Jiang, P. Wang, X. Gong, D. Xu, and Z. Wang. Unified implicit neural stylization. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XV , pages 636-654. Springer, 2022.

J. Fang, T. Yi, X. Wang, L. Xie, X. Zhang, W. Liu, M. Nießner, and Q. Tian. Fast dynamic radiance fields with time-aware neural voxels. In SIGGRAPH Asia 2022 Conference Papers , 2022.

C. Gao, A. Saraf, J. Kopf, and J.-B. Huang. Dynamic view synthesis from dynamic monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5712-5721, 2021.

H. Gao, R. Li, S. Tulsiani, B. Russell, and A. Kanazawa. Dynamic novel-view synthesis: A reality check. In NeurIPS , 2022.

S. J. Garbin, M. Kowalski, M. Johnson, J. Shotton, and J. Valentin. Fastnerf: High-fidelity neural rendering at 200fps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14346-14355, 2021.

Y.-C. Guo, D. Kang, L. Bao, Y. He, and S.-H. Zhang. Nerfren: Neural radiance fields with reflections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18409-18418, June 2022.

P. Hedman, P. P. Srinivasan, B. Mildenhall, J. T. Barron, and P. Debevec. Baking neural radiance fields for real-time view synthesis. ICCV , 2021.

G. Hinton, O. Vinyals, and J. Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

T. Hu, S. Liu, Y. Chen, T. Shen, and J. Jia. Efficientnerf efficient neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 12902-12911, June 2022.

W. Jang and L. Agapito. Codenerf: Disentangled neural radiance fields for object categories. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 12949-12958, October 2021.

Y. Jeong, S. Ahn, A. A. Christopehr Choy, M. Cho, and J. Park. Self-calibrating neural radiance fields. In ICCV , 2021.

Z. Jiakai, L. Xinhang, Y. Xinyi, Z. Fuqiang, Z. Yanshun, W. Minye, Z. Yingliang, X. Lan, and Y. Jingyi. Editable free-viewpoint video using a layered neural representation. In ACM SIGGRAPH , 2021.

Y. Jiang, P. Hedman, B. Mildenhall, D. Xu, J. T. Barron, Z. Wang, and T. Xue. Alignerf: High-fidelity neural radiance fields via alignment-aware training. arXiv preprint arXiv:2211.09682 , 2022.

T. Kanade, P. Rander, and P. Narayanan. Virtualized reality: Constructing virtual worlds from real scenes. IEEE multimedia , 4(1):34-47, 1997.

A. Kendall and Y. Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advances in neural information processing systems , 30, 2017.

B. Kerbl, G. Kopanas, T. Leimkühler, and G. Drettakis. 3d gaussian splatting for real-time radiance field rendering. ACM Transactions on Graphics , 42(4), July 2023. URL https://repo-sam.inria.fr/ fungraph/3d-gaussian-splatting/ .

S. Klenk, L. Koestler, D. Scaramuzza, and D. Cremers. E-nerf: Neural radiance fields from a moving event camera. IEEE Robotics and Automation Letters , 2023.

S. Kobayashi, E. Matsumoto, and V. Sitzmann. Decomposing nerf for editing via feature field distillation. In Advances in Neural Information Processing Systems , volume 35, 2022. URL https://arxiv.org/ pdf/2205.15585.pdf .

L. Li, Z. Shen, Z. Wang, L. Shen, and P. Tan. Streaming radiance fields for 3d video synthesis. arXiv preprint arXiv:2210.14831 , 2022.

R. Li, J. Tanke, M. Vo, M. Zollhofer, J. Gall, A. Kanazawa, and C. Lassner. Tava: Template-free animatable volumetric actors. 2022.

T. Li, M. Slavcheva, M. Zollhoefer, S. Green, C. Lassner, C. Kim, T. Schmidt, S. Lovegrove, M. Goesele, R. Newcombe, et al. Neural 3d video synthesis from multi-view video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5521-5531, 2022.

Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , 2021.

Z. Li, S. Niklaus, N. Snavely, and O. Wang. Neural scene flow fields for space-time view synthesis of dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 6498-6508, 2021.

C.-H. Lin, W.-C. Ma, A. Torralba, and S. Lucey. Barf: Bundle-adjusting neural radiance fields. In IEEE International Conference on Computer Vision (ICCV) , 2021.

D. B. Lindell, J. N. Martel, and G. Wetzstein. Autoint: Automatic integration for fast neural volume rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14556-14565, 2021.

H.-K. Liu, I.-C. Shen, and B.-Y. Chen. Nerf-in: Free-form nerf inpainting with rgb-d priors. arxiv preprint arXiv:2206.04901 , 2022.

J.-W. Liu, Y.-P. Cao, W. Mao, W. Zhang, D. J. Zhang, J. Keppo, Y. Shan, X. Qie, and M. Z. Shou. Devrf: Fast deformable voxel radiance fields for dynamic scenes. arXiv preprint arXiv:2205.15723 , 2022.

L. Liu, J. Gu, K. Zaw Lin, T.-S. Chua, and C. Theobalt. Neural sparse voxel fields. Advances in Neural Information Processing Systems , 33:15651-15663, 2020.

S. Liu, X. Zhang, Z. Zhang, R. Zhang, J.-Y. Zhu, and B. Russell. Editing conditional radiance fields, 2021.

Y.-L. Liu, C. Gao, A. Meuleman, H.-Y. Tseng, A. Saraf, C. Kim, Y.-Y. Chuang, J. Kopf, and J.-B. Huang. Robust dynamic radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023.

S. Lombardi, T. Simon, J. Saragih, G. Schwartz, A. Lehrmann, and Y. Sheikh. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 , 2019.

S. Lombardi, T. Simon, G. Schwartz, M. Zollhoefer, Y. Sheikh, and J. Saragih. Mixture of volumetric primitives for efficient neural rendering, 2021.

L. Ma, X. Li, J. Liao, Q. Zhang, X. Wang, J. Wang, and P. V. Sander. Deblur-nerf: Neural radiance fields from blurry images. arXiv preprint arXiv:2111.14292 , 2021.

R. Martin-Brualla, N. Radwan, M. S. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. Nerf in the wild: Neural radiance fields for unconstrained photo collections. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 7210-7219, 2021.

R. Martin-Brualla, N. Radwan, M. S. M. Sajjadi, J. T. Barron, A. Dosovitskiy, and D. Duckworth. NeRF in the Wild: Neural Radiance Fields for Unconstrained Photo Collections. In CVPR , 2021.

Q. Meng, A. Chen, H. Luo, M. Wu, H. Su, L. Xu, X. He, and J. Yu. Gnerf: Gan-based neural radiance field without posed camera. arXiv preprint arXiv:2103.15606 , 2021.

B. Mildenhall, P. Hedman, R. Martin-Brualla, P. P. Srinivasan, and J. T. Barron. NeRF in the dark: High dynamic range view synthesis from noisy raw images. arXiv , 2021.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM , 65(1):99-106, 2021.

T. Müller, A. Evans, C. Schied, and A. Keller. Instant neural graphics primitives with a multiresolution hash encoding. arXiv preprint arXiv:2201.05989 , 2022.

M. Niemeyer and A. Geiger. Campari: Camera-aware decomposed generative neural radiance fields, 2021.

M. Niemeyer, J. T. Barron, B. Mildenhall, M. S. M. Sajjadi, A. Geiger, and N. Radwan. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR) , 2022.

M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3d reconstruction at scale using voxel hashing. ACM Trans. Graph. , 32(6):169:1-169:11, 2013. doi: 10.1145/2508363.2508374. URL https://doi.org/10.1145/2508363.2508374 .

X. Pan, X. Xu, C. C. Loy, C. Theobalt, and B. Dai. A shading-guided generative implicit model for shapeaccurate 3d-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , 2021.

K. Park, U. Sinha, J. T. Barron, S. Bouaziz, D. B. Goldman, S. M. Seitz, and R. Martin-Brualla. Nerfies: Deformable neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5865-5874, 2021.

K. Park, U. Sinha, P. Hedman, J. T. Barron, S. Bouaziz, D. B. Goldman, R. Martin-Brualla, and S. M. Seitz. Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields. arXiv preprint arXiv:2106.13228 , 2021.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Köpf, E. Z. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In H. M. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. B. Fox, and

R. Garnett, editors, Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada , pages 8024-8035, 2019. URL https://proceedings.neurips.cc/paper/2019/hash/ bdbca288fee7f92f2bfa9f7012727740-Abstract.html .

S. Peng, Y. Zhang, Y. Xu, Q. Wang, Q. Shuai, H. Bao, and X. Zhou. Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans. In CVPR , 2021.

S. Peng, Y. Yan, Q. Shuai, H. Bao, and X. Zhou. Representing volumetric videos as dynamic mlp maps. In CVPR , 2023.

A. Pumarola, E. Corona, G. Pons-Moll, and F. Moreno-Noguer. D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10318-10327, 2021.

D. Rebain, W. Jiang, S. Yazdani, K. Li, K. M. Yi, and A. Tagliasacchi. Derf: Decomposed radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14153-14161, 2021.

C. Reiser, S. Peng, Y. Liao, and A. Geiger. Kilonerf: Speeding up neural radiance fields with thousands of tiny mlps. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14335-14345, 2021.

C. Reiser, R. Szeliski, D. Verbin, P. P. Srinivasan, B. Mildenhall, A. Geiger, J. T. Barron, and P. Hedman. Merf: Memory-efficient radiance fields for real-time view synthesis in unbounded scenes. arXiv preprint arXiv:2302.12249 , 2023.

Sara Fridovich-Keil and Giacomo Meanti, F. R. Warburg, B. Recht, and A. Kanazawa. K-planes: Explicit radiance fields in space, time, and appearance. In CVPR , 2023.

K. Schwarz, Y. Liao, M. Niemeyer, and A. Geiger. Graf: Generative radiance fields for 3D-aware image synthesis. In Advances in Neural Information Processing Systems (NeurIPS) , volume 33, 2020.

R. Shao, Z. Zheng, H. Tu, B. Liu, H. Zhang, and Y. Liu. Tensor4d: Efficient neural 4d decomposition for high-fidelity dynamic reconstruction and rendering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , 2023.

L. Song, A. Chen, Z. Li, Z. Chen, L. Chen, J. Yuan, Y. Xu, and A. Geiger. Nerfplayer: A streamable dynamic scene representation with decomposed neural radiance fields. arXiv preprint arXiv:2210.15947 , 2022.

P. Srinivasan, B. Deng, X. Zhang, M. Tancik, B. Mildenhall, and J. T. Barron. NeRV: Neural reflectance and visibility fields for relighting and view synthesis. https://arxiv.org/abs/2012.03927 , 2020.

S.-Y. Su, F. Yu, M. Zollhöfer, and H. Rhodin. A-nerf: Articulated neural radiance fields for learning human shape, appearance, and pose. In Advances in Neural Information Processing Systems , 2021.

E. Sucar, S. Liu, J. Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-time. In arXiv preprint arXiv:2103.12352 , 2021.

C. Sun, M. Sun, and H.-T. Chen. Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 5459-5469, 2022.

M. Tancik, B. Mildenhall, T. Wang, D. Schmidt, P. Srinivasan, J. T. Barron, and R. Ng. Learned initializations for optimizing coordinate-based neural representations. https://arxiv.org/abs/2012.02189 , 2020.

M. Tancik, V. Casser, X. Yan, S. Pradhan, B. Mildenhall, P. P. Srinivasan, J. T. Barron, and H. Kretzschmar. Block-nerf: Scalable large scene neural view synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 8248-8258, 2022.

E. Tretschk, A. Tewari, V. Golyanik, M. Zollhöfer, C. Lassner, and C. Theobalt. Non-rigid neural radiance fields: Reconstruction and novel view synthesis of a dynamic scene from monocular video. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12959-12970, 2021.

F. Wang, S. Tan, X. Li, Z. Tian, and H. Liu. Mixed neural voxels for fast multi-view video synthesis. arXiv preprint arXiv:2212.00190 , 2022.

H. Wang, J. Ren, Z. Huang, K. Olszewski, M. Chai, Y. Fu, and S. Tulyakov. R2l: Distilling neural radiance field to neural light field for efficient novel view synthesis. In ECCV , 2022.

L. Wang, J. Zhang, X. Liu, F. Zhao, Y. Zhang, Y. Zhang, M. Wu, J. Yu, and L. Xu. Fourier plenoctrees for dynamic radiance field rendering in real-time. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13524-13534, 2022.

P. Wang, Y. Liu, Z. Chen, L. Liu, Z. Liu, T. Komura, C. Theobalt, and W. Wang. F2-nerf: Fast neural radiance field training with free camera trajectories. CVPR , 2023.

Q. Wang, Z. Wang, K. Genova, P. Srinivasan, H. Zhou, J. T. Barron, R. Martin-Brualla, N. Snavely, and T. Funkhouser. Ibrnet: Learning multi-view image-based rendering. arXiv preprint arXiv:2102.13090 , 2021.

T. Wang, B. Zhang, T. Zhang, S. Gu, J. Bao, T. Baltrusaitis, J. Shen, D. Chen, F. Wen, Q. Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. arXiv preprint arXiv:2212.06135 , 2022.

Z. Wang, S. Wu, W. Xie, M. Chen, and V. A. Prisacariu. NeRF --: Neural radiance fields without known camera parameters. arXiv preprint arXiv:2102.07064 , 2021.

Z. Wang, L. Li, Z. Shen, L. Shen, and L. Bo. 4k-nerf: High fidelity neural radiance fields at ultra high resolutions. arXiv preprint arXiv:2212.04701 , 2022.

W. Xian, J.-B. Huang, J. Kopf, and C. Kim. Space-time neural irradiance fields for free-viewpoint video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 9421-9431, 2021.

D. Xu, Y. Jiang, P. Wang, Z. Fan, H. Shi, and Z. Wang. Sinnerf: Training neural radiance fields on complex scenes from a single image. 2022.

L. Yariv, P. Hedman, C. Reiser, D. Verbin, P. P. Srinivasan, R. Szeliski, J. T. Barron, and B. Mildenhall. Bakedsdf: Meshing neural sdfs for real-time view synthesis. arXiv preprint arXiv:2302.14859 , 2023.

L. Yen-Chen, P. Florence, J. T. Barron, A. Rodriguez, P. Isola, and T.-Y. Lin. iNeRF: Inverting neural radiance fields for pose estimation. https://arxiv.org/abs/2012.05877 , 2020.

Z.-X. Yin, J. Qiu, M.-M. Cheng, and B. Ren. Multi-space neural radiance fields. 2023.

A. Yu, S. Fridovich-Keil, M. Tancik, Q. Chen, B. Recht, and A. Kanazawa. Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131 , 2021.

A. Yu, R. Li, M. Tancik, H. Li, R. Ng, and A. Kanazawa. Plenoctrees for real-time rendering of neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 5752-5761, 2021.

A. Yu, V. Ye, M. Tancik, and A. Kanazawa. pixelNeRF: Neural radiance fields from one or few images. In CVPR , 2021.

W. Yuan, Z. Lv, T. Schmidt, and S. Lovegrove. Star: Self-supervised tracking and reconstruction of rigid objects in motion with neural rendering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13144-13152, 2021.

K. Zhang, G. Riegler, N. Snavely, and V. Koltun. Nerf++: Analyzing and improving neural radiance fields. arXiv preprint arXiv:2010.07492 , 2020.

R. Zhang, P. Isola, A. A. Efros, E. Shechtman, and O. Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

F. Zhao, W. Yang, J. Zhang, P. Lin, Y. Zhang, J. Yu, and L. Xu. Humannerf: Efficiently generated human radiance field from sparse inputs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 7743-7753, June 2022.

Z. Zhu, S. Peng, V. Larsson, W. Xu, H. Bao, Z. Cui, M. R. Oswald, and M. Pollefeys. Nice-slam: Neural implicit scalable encoding for slam, 2021.

C. L. Zitnick, S. B. Kang, M. Uyttendaele, S. Winder, and R. Szeliski. High-quality video view interpolation using a layered representation. ACM transactions on graphics (TOG) , 23(3):600-608, 2004.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the masked space-time hash encoding to efficiently reconstruct dynamic scenes. The insights behind the paper is that: most part of the scene is static, simply modelling such static parts can dramatically increase the probability of hash collisions; while increasing the hash table entries requires more memory. To solve these issues, this paper proposes to decouple the scene into a static part and a dynamic part, where the two parts are jointly learned through an uncertainty mask. The uncertainty mask is modeled by an individual voxel grid, on which each voxel stores an value of the uncertainty field. To bridge the gap between uncertainties and masks, a neural estimator is adopted to approximate the mutual information. Experiments are conducted on the Plenoptic Video Dataset, the Google Immersive Dataset, and the self-collected time-synchronized multi-view videos. As shown in the quantitative results, MSTH surpasses state-of-the-art dynamic NeRF methods (HexPlane, K-Planes, etc) in terms of both the reconstruction quality and training/rendering time. Ablation studies also shows the effectiveness of the proposed masked space-time hash encoding.\n\nSTRENGTHS:\nThis is a technically very solid paper. Compared to previous methods, the proposed method can reconstruct dynamic scenes with higher quality while highly reduced the training time. I like the idea of using uncertainty to mask the static part and dynamic part in a scene instead of simply a combination of 3D voxel grid and a 4D voxel grid. The experiments are exhaustive and highly support the effectiveness of the proposed method.\n\nWEAKNESSES:\n(1) In Figure 3 (c), it seems some static parts have high uncertainty values (the top-left part and the middle-right part)?\n\nQUESTIONS:\n- (1)  Is Figure 3(c) revealing that the uncertainty does not accurately model the static part and dynamic part?\n- (2) In Section 4.3, the authors proposed two variants for ablation study: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Actually, for the second variant, I'm not quite sure whether the authors split the scene into a static part or a dynamic part using additional models (for example, using the **segment anything model (SAM)** to split a non-rigid object/human from the static background), and then separately model them. If it is not, I wonder would it better to separately model the dynamic part and static part with some pre-computed masks instead of jointly learn the uncertainty during training?\n\nLIMITATIONS:\nThe authors address the major limitations of existing dynamic NeRF methods, e.g. the efficiency and rendering quality. Some other limitations, such as reconstructing monocular dynamic scenes, blurry scenes, etc, are mentioned in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents MSTH, a method that efficiently reconstructs dynamic 3D scenes from multi-view or monocular videos. The proposed solution uses a space-time hash encoding, a prediction of masks, and uncertainty values that help a method to identify 3D points that belong to moving objects. The intuition is that this masking and uncertainty will help a method identify static points which are easy to reconstruct but also handle well the moving points. Lastly, the formulation of the method allows it to render in fast manner also preserving the size of the model. The paper presents experiments on three real-datasets: Plenoptic Video Dataset, Google Immersive Dataset, and a newly introduced dataset. The presented experiments show that the MSTH improves the PSNR while keeping both the training time and inference time short as well as maintaining a constant size model.\n\nSTRENGTHS:\nIn sum, I find the paper to be well executed and explained. Overall, I think the problem of reconstructing dynamic scenes is a challenging but important problem to solve. In detail here are the strengths I find:\n\nS1. I find the use of masking to identify the static and dynamic points an interesting and simple idea. I think the idea has good intuitive rationale and it should be effective.\n\nS2. The inclusion of uncertainty in the formulation I think is an important example of why uncertainty is very important in machine learning. I support the idea of combining uncertainty with the mask prediction as a mean to improve results.\n\nS3. The experiments use real-data and challenging scenes to showcase the benefits of the proposed approach. Overall, I think the experiments are well executed and highlight the benefits of the MSTH.\n\nS4. The clarity and description of MSTH is good. Overall, I think the proposed method should be easy to reproduced given the level of clarity of the narrative.\n\nWEAKNESSES:\nOverall, I find the paper to be well executed and clear. I only have one weakness to point out:\n\nW1. I think the paper is lacking a more in-depth discussion about the newly created dataset to test MSTH. Whil I understand that the algorithm is important, I also think the data part is as important as the algorithm. At the end of the day, the data is required and crucial to solve many problems in the era of deep learning. Thus, I suggest that a final version of the paper includes an extended discussion of the dataset curated by the authors.\n\nWhile I like the approach and explanation of MSTH, I do have some minor concerns:\n\nW2. The uncertainty described in the method in a way breaks with the intuitive understanding that an uncertainty value corresponds to a confidence value ranging from 0-1 and could be interpreted as a probability of correctness. However, the \"uncertainty\" introduced in Section 3.3 deviates from this common understanding. I think the paper should explicitly state a concrete meaning of uncertainty to clarify that the common understanding of uncertainty does not apply to this narrative. The main reason I bring this up is because I think this can confuse readers in the future.\n\nW3. In several parts of the narrative (e.g., lines 188 and 186) the paper uses the term \"dynamics of a point\". I find this term to be confusing since in many fields (e.g., robotics or control) the dynamics typically refer to a model that describes how the state of something changes over time. I think the paper should clarify what this exactly means to clear any misconception of this term.\n\nW4. Missing definition of different terms: 1) $\\tilde{m}(\\cdot)$ is never defined in the text; 2) what does \"large $m$\" means in line 168? Isn't $m$ the learned mask? Or is it a value?\n\n----\nPost Rebuttal\n\nAfter the rebuttal, all my concerns were clarified and I still think this is a good contribution. Thus, I maintain my rating.\n\nQUESTIONS:\nPlease see Weaknesses section.\n\nLIMITATIONS:\nI think the paper clearly states the limitations of the approach.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper present a method for efficient dynamic scene reconstruction. They represent the dynamic scene with a weighted combination of a 3D hash encoding (for static part) and a 4D hash encoding (for dynamic region). The weight is learnable and can be represented by a multi-resolution hash table or a 3D voxel grid. For each query 3D point, they can then get a weighted feature encoding from the hash tables. The photometric loss is then used to supervise the training. \n\nTo better supervise the learnable mask/weight representation, they further propose an uncertainty guided mask learning and exploit the mutual correlation between \"3D point uncertainty\" and \"3D point mask\" to supervise the learning of the mask representation. \n\nThe experimental evaluations are conducted on two public datasets and a self-collected dataset. The experimental results demonstrate the effectiveness of the proposed method.\n\nSTRENGTHS:\n1: The paper presents a novel method to improve both the reconstruction quality and efficiency for dynamic scenes. The experimentral results demonstrate its effectiveness. \n\n2: The paper is well written.\n\nWEAKNESSES:\nI do not find severe flaws of the paper and would like to discuss with other reviewers if necessary.\n\nQUESTIONS:\nThe mask representation is not time dependent and the uncertainty supervision is based on the inconsistency between the rendered images & the real captured images. I have following two questions:\n\n1: Since the 3D scene is dynamic, the mask(static/dynamic) for each 3D position should also depend on time. Why you choose to use a time-independent representation. \n\n2: Is it like that as long as the 3D point being occupied by a dynamic object (even for partial frames), it will also be predicted as dynamic (which would have a high uncertainty from Eq. 7)? If so, will your method be able to work well if the dynamic object occupy a larger portion of the image and sweep over most of the 3D space across time. To what extent it can perform well?\n\nLIMITATIONS:\nIf Question 2 is right, please address the corresponding limitation in Section 5 of the main paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

comments: SUMMARY:
This paper tries to address the challenge of efficiently representing 3D dynamic scenes. It proposes a decoupled representation that uses separate neural implicit representations for dynamic and static 3D points. This approach can reduce hash collision and save the storage of the multi-level hash feature grids.

STRENGTHS:
1. An automatic learning procedure to detect dynamic points in the 3D scene. This is achieved by integrating Bayesian learning to treat dynamic points as noise and reduce their weights in the photo-consistency loss.
2. A separate 3D and 4D hash feature grids to save the storage.  
3. The optimization of mutual information between uncertainty and mask is demonstrated to be beneficial to the rendering quality.

WEAKNESSES:
1. The concept of representing scenes with separate static and dynamic parts has been investigated in NeRFPlayer or NeRF in the wild.  Integrating Bayesian learning to predict the dynamic points is also not new. 
2. Lack quantitative ablation study on the mutual information optimization, which weaken the contribution of this step. 
3. Inadequate references. There are missing references related to neural implicit representation for large scale and dynamic scenes:
 Xiuchao Wu et al., Scalable Neural Indoor Scene Rendering, SIGGRAPH 2022
 Haithem Turki et al., Mega-NERF: Scalable Construction of Large-Scale NeRFs for Virtual Fly-Throughs, CVPR 2022
 Liao Wagn et al., Fourier PlenOctrees for Dynamic Radiance Field Rendering in Real-time, CVPR 2022 Oral

QUESTIONS:
In Fig.6, what is the meaning of each row?

LIMITATIONS:
The proposed method is most effective for scenes with a significant amount of static parts. For scenes with a small amount of static parts, the storage savings will be less significant.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05