SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In practice, the model can learn a reasonable mask  $m$  even without the uncertainty guidance. However, this will make the learned mask very noisy and tend to learn a "middle value" in [0, 1], which will make the static points still have a relatively large dynamic weight to modify the 4D table. Fig. 3 visualize the learned mask with or without uncertainty guidance. With uncertainty guidance, the mask will tend to be binarized and the static parts are with very low dynamic weight. For the mutual information constraint, we find it will make the distribution of  $m$  towards a Bernoulli distribution which is helpful for reducing hash collision and accelerating rendering speed. Without the constraint, the model tends to learn more from the 3D hash table only even for the dynamic point. This will make the failing captures of some dynamic voxels with transient changes. As a result, the uncertainty guidance will help learn a finer detail, which will be shown in the ablation part.
3.4 Ray Sampling
For a natural video, a significant portion of the scene is usually static or exhibits only minor changes in radiance at a specific time across the entire video. Uniformly sampling the space-time rays causes an imbalance between the static and dynamic observations. Therefore, we sample the space-time queries according to the quantification of dynamics. Specifically, we sample a space-time ray  $(\boldsymbol{r},t)$  according to a pre-computed probability  $P(\boldsymbol{r},t)$ . We decompose the sampling process into two sub-process: spatial sampling and temporal sampling. Formally, we decompose  $P(\boldsymbol{r},t)$  as  $P(\boldsymbol{r},t) = P(t|\boldsymbol{r})P(\boldsymbol{r})$  which forms a simple Markov process that first sample the space ray according to  $P(\boldsymbol{r})$  then sample the time step by  $P(t|\boldsymbol{r})$ .
Plenoptic Video Dataset
Immersive Dataset
Campus Dataset
Figure 4: Qualitative results on different datasets of our method. We visualize four novel views in each column, three for RGB and one for depth. For other scenes, we visualize them in the Appendix, and the full spiral videos are presented in our supplemental material.
In practice, we build  $P(\boldsymbol{r})$  as the softmax of temporal standard deviation  $\operatorname{std}(r)$  of the corresponding pixel intensities  $\{C(r,t)\}_{t}$ :
$$
P (\boldsymbol {r}) \propto \exp \left(\operatorname {s t d} (\boldsymbol {r}) / \tau_ {1}\right), \tag {10}
$$
where  $\tau_{1}$  is the temperature parameter. We set a higher temperature for a softer probability distribution [27]. For temporal sampling, we follow [42] to compute the weight according to the residual difference of its color to the global median value across time  $\bar{C} (\boldsymbol {r})$  with temperature  $\tau_{2}$ :
$$
P (t | \boldsymbol {r}) \propto \exp \left(| C (\boldsymbol {r}, t) - \bar {C} (\boldsymbol {r}) | / \tau_ {2}\right). \tag {11}
$$
To improve the sampling efficiency, we utilize the proposal sampler [4] that models density at a coarse granularity with a much smaller field, thereby generating more accurate samples that align with the actual density distribution with minor overhead. In our implementation, we use one layer of proposal net to sample 128 points. For the mask, we use a non-cache voxel grid with 128 spatial resolution. To encourage the separation of the static and the dynamic, we utilize a mask loss that aims at generating sparse dynamic voxels by constraining the mask to be close at 1. We also adopt distortion loss [4] with  $\lambda_{dist} = 2e - 2$ . For uncertainty loss, we set  $\gamma = 3e - 4$  and  $\lambda = 3e - 5$ . We find a small coefficient of mutual information estimator  $\gamma$  will have a large impact on the gradients, which is consistent with the observations of [6]. For the hash grid, we implement a CUDA extension based on PyTorch [65] to support the rectangular voxel grid required by the proposed method. For the complete experiment setting and model hyper-parameters, please refer to the Appendix.
For validating the performances of the proposed method, we conduct experiments on two public datasets and our collected dataset: (1) The Plenoptic Video Dataset [40], which consists of 6 publicly accessible scenes: coffee-martini, flame-salmon, cook-spinach, cut-roasted-beef, flame-steam, and sear-steam. Each scene contains 19 videos with different camera views. The dataset contains some challenging scenes, including objects with topology changes, objects with volumetric effects, various lighting conditions, etc. (2) Google Immersive Dataset [9]: The Google Immersive dataset contains light field videos of different indoor and outdoor environments captured by a time-synchronized 46-fisheye camera array. We use the same 7 scenes (Welder, Flames, Truck, Exhibit, Alexa Meade
Table 1: Quantitative results on Plenoptic Video dataset [40]. We report the average metrics and compare them with other state-of-the-art methods. Our method achieves non-trivial performance improvements on all metrics. * denotes the DyNeRF setting which only reports results on the flame-salmon scene. † denotes the HexPlane setting that removes the coffee-martini scene to calculate average metrics. We report the per-scene metrics in the Appendix.
Model	PSNR↑	D-SSIM↓	LPIPS↓	Training Time↓	Rendering FPS ↑	Storage ↓
DyNeRF* [40]	29.58	0.020	0.099	1344 h	< 0.01	28MB
Ours*	29.92	0.020	0.063	20 min	15	135MB
NeRFPlayer [75]	30.69	0.034	0.111	6 h	0.05	-
HyperReel [1]	31.10	0.036	0.096	9 h	2.0	360MB
Mix Voxels [83]	30.80	0.020	0.126	80 min	16.7	500MB
K-Planes [72]	31.63	0.018	-	108 min	-	-
Ours	32.37	0.015	0.056	20 min	15	135MB
HexPlane [10] †	31.705	0.014	0.075	12 h	-	200MB
Ours†	33.099	0.013	0.051	20 min	15	135MB
Exhibit, Face Paint 1, Face Paint 2) as NeRFPlayer [75] for evaluation. (3) To validate the robustness of our method on more complex in-the-wild scenarios, we collect six time-synchronized multi-view videos including more realistic observations such as pedestrians, moving cars, and grasses with people playing. We named the collected dataset as Campus Dataset. The Campus dataset is much more difficult than the above two curated ones in the movement complexities and dynamic areas. For detail on the dataset, please see our Appendix. For the above three multi-view datasets, we follow the experiment setting in [40] that employs 18 views for training and 1 view for evaluation. To quantitatively evaluate the rendering quality on novel views, we measure PSNR, DSSIM, and LPIPS [101] on the test views. We follow the setting of [40] to evaluate our model frame by frame. Our method is also applicable to monocular videos. To validate the reconstruction quality with monocular input, we conduct experiments on D-NeRF [68] dataset, which contains eight videos of varying duration, ranging from 50 frames to 200 frames per video. There is only one single training image for each time step. For evaluation, we follow the common setting from [68, 72, 10].
Multi-view Dynamic Scenes. We reconstruct the dynamic scenes from multi-view time-synchronized observations on the two public multi-view video datasets and our collected Campus dataset. The quantitative results and comparisons are presented in Tab. 1 and Tab. 2. For the Plenoptic Video dataset, our method surpasses previous state-of-the-art methods by a non-trivial margin, with a 0.7 to 1.4 PSNR gains, and  $>30\%$  improvements on the perceptual metric LPIPS. For training, we cost at most  $1/5$  training time of other methods, achieving a speedup of  $5 - 36$  compared with other fast reconstruction algorithms and a 4000 speedup compared with DyNeRF [40]. Besides, we also keep a compact model size with only 135MB of memory occupancy, showing the advantages of the masking strategy which can avoid a large number of hash collisions and make the dynamic hash table size smaller enough. For the Google Immersive dataset, Tab. 2 also shows consistently non-trivial improvements in both training efficiency and reconstruction quality.
Fig. 2 visualizes the qualitative comparisons of our method to other state-of-the-art methods, from which we can observe that our method can capture finer motion details than other methods. For example, MSTH can well reconstruct the fire gun with distinct boundaries and specular reflection, the mark of the hat, and the stripe of the salmon. The Splashing sparks can also be accurately captured. We provide the representative novel-view rendering results of our method in Fig. 4.
Monocular Dynamic Scenes. We present the quantitative and qualitative results of our proposed method on the D-NeRF dataset in Tab. 3 and Fig. 5, accompanied by a comparative analysis with related approaches. MSTH outperforms all other methods in terms of SSIM and LPIPS, including methods that focus on dynamic scenes (K-Planes [72] and HexPlane [10]) and TiNeuVox, the current state-of-the-art method for monocular dynamic novel view synthesis. The proposed MSTH achieves superior results without any assumption about the underlying field which demonstrates that our method could be easily extended to accommodate scenarios with varying complexity and dynamics.
Figure 5: Qualitative results on D-NeRF dataset [68]. The visualized images are rendered from test views. Complete per-scene multi-view images are shown in the Appendix.
Table 2: Quantitative result on Google Immersive dataset [9] and the proposed Campus dataset. Check out the Appendix for detailed comparison and visualization.
Model	PSNR↑	SSIM↑	LPIPS↓	Training Time ↓
Google Immersive Video Dataset
NeRFPlayer [75]	26.6	0.870	0.1931	6 hrs
HyperReel [1]	28.8	0.874	0.193	2.7 hrs
Ours	29.6	0.950	0.0929	20 min
Campus Dataset
Ours	20.9	0.722	0.241	20 min
Table 3: Quantitative comparison with related work on D-NeRF dataset [68]. Per-scene metrics are shown in the Appendix.
Model	PSNR↑	SSIM↑	LPIPS↓
T-NeRF [68]	29.51	0.95	0.08
D-NeRF [68]	30.5	0.95	0.07
TiNeuVox-B [21]	32.67	0.97	0.04
HexPlane [10]	31.04	0.97	0.04
K-Planes [72]	30.84	0.96	-
Ours	31.34	0.98	0.02
Ablation on the Masked Hash Encoding. To evaluate the effect of decomposing a 4D dynamic radiance field into the proposed masked space-time hash encoding, we propose two variants as comparisons: (1) A pure 4D hash encoding. (2) A simple decomposition which is an addition of a 3D hash table and a 4D hash table. Fig. 6 (a) and (b) visualize the qualitative comparisons. From Fig. 6 (a) we can observe that only using the 4D hash table will generate blurry rendering with a 3 PSNR drop. With a simple addition method, the reconstruction quality is improved compared with only a 4D hash table, while the dynamic regions are not captured well. Fig. 6 (b) and (c) show the comparisons of MSTH with the addition. MSTH improves the LPIPS by  $20\%$  compared with the addition method.
Ablation on the uncertainty-based guidance. We visualize the learned mask with or without uncertainty-based guidance, Fig. 3 shows the comparison. The learned mask with uncertainty loss will distinguish the static and dynamic parts more clearly, with less noisy judgment, leading to an assertive distribution on the mask, which is beneficial to avoid hash collisions. Fig. 6 (d) visualize the qualitative comparisons. MSTH with uncertainty loss performs better on the motion details.
(c)
Figure 6: Qualitative comparisons for different ablations. Zoom in for a better inspection.
Limitations. MSTH tends to generate unsatisfying results when detailed dynamic information is insufficient, especially in monocular dynamic scenes, since it relies solely on the input images. When the input is blurred, occluded, or incomplete, the method may struggle to infer the motion information, leading to artifacts such as ghosting, flickering, or misalignment, which may degrade the visual quality. Further research is needed to develop advanced methods that handle complex scenes, motion dynamics and integrate multiple information sources to enhance synthesis accuracy and robustness.
Conclusion. In this paper, we propose a new method to reconstruct dynamic 3D scenes in a time- and space-efficient way. We decouple the representations of the dynamic radiance field into a time-invariant 3D hash encoding and a time-variant 4D hash encoding. With an uncertainty-guided mask as weight, our algorithm can avoid a large number of hash collisions brought about by the additional time dimension. We conduct extensive experiments to validate our method and achieve state-of-the-art performances with only 20 minutes of training. Besides, we collect a complex in-the-wild multi-view video dataset for evaluating the robustness of our approach on more realistic dynamic
scenes, including many daily activities. We hope our method can serve as a fast and lightweight baseline to reconstruct dynamic 3D scenes, which may derive many valuable applications such as interactively free-viewpoint control for movies, cinematic effects, novel view replays for sporting events, and other VR/AR applications.
This work was supported in part by the National Natural Science Fund for Distinguished Young Scholars under Grant 62025304. Thank Xueping Shi for giving helpful discussion.