SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Feng Wang*1 Zilong Chen*1 Guokang Wang1 Yafei Song2 Huaping Liu†1
<sup>1</sup>Beijing National Research Center for Information Science and Technology(BNRist), Department of Computer Science and Technology, Tsinghua University
2 Alibaba Group
wang-f20@mails.tsinghua.edu.cn, hpliu@tsinghua.edu.cn
In this paper, we propose the Masked Space-Time Hash encoding (MSTH), a novel method for efficiently reconstructing dynamic 3D scenes from multi-view or monocular videos. Based on the observation that dynamic scenes often contain substantial static areas that result in redundancy in storage and computations, MSTH represents a dynamic scene as a weighted combination of a 3D hash encoding and a 4D hash encoding. The weights for the two components are represented by a learnable mask which is guided by an uncertainty-based objective to reflect the spatial and temporal importance of each 3D position. With this design, our method can reduce the hash collision rate by avoiding redundant queries and modifications on static areas, making it feasible to represent a large number of space-time voxels by hash tables with small size. Besides, without the requirements to fit the large numbers of temporally redundant features independently, our method is easier to optimize and converge rapidly with only twenty minutes of training for a 300-frame dynamic scene. As a result, MSTH obtains consistently better results than previous methods with only 20 minutes of training time and  $130\mathrm{MB}$  of memory storage. Code is available at https://github.com/masked-spacetime hashing/msth.
Neural radiance fields [57, 96, 79, 58, 14] have achieved great success in reconstructing static 3D scenes. The learned volumetric representations can produce photo-realistic rendering for novel views. However, for dynamic scenes, the advancements lag behind that of static scenes due to the inherent complexities caused by the additional time dimension. Specifically, reconstructing dynamic scenes requires more time, memory footprint, and additional temporal consistency compared to static scenes. These factors have resulted in unsatisfactory rendering qualities. Recently, many works [40, 38, 75, 83, 1, 10, 72] have made great progress in dynamic scene reconstruction, improving the efficiency and effectiveness of recon
Figure 1: Performance comparison. We compare the PSNR and training time on two public benchmarks. Our method surpasses other methods by a non-trivial margin with only  $20\mathrm{m}$  of training.  $\dagger$  denotes the HexPlane [10] setting which removes the coffee-martini scene.
DyNeRF[40] 1344h
HexPlane[10] 12h
Ours 20min
NeRFPlayer[75] 6h
HyperReel[1] 2.7h
DyNeRF[40] 1344h
Figure 2: Qualitative comparisons to state-of-the-art methods [40, 1, 75, 10]. We visualize three scenes: flame salmon, horse, and welder from Plenoptic Video dataset [40] and Google Immersive dataset [9]. Some key patches are zoomed in for better inspection. Our method performs better in reconstructing details, such as the stripes of the salmon, the facial features, and the splashing sparks.
for improvement in many aspects such as rendering quality, motion coherence, training and rendering speed, and memory usage.
This paper aims to develop a method for reconstructing dynamic scenes with high rendering quality in a space- and time-efficient manner. To achieve this goal, we base our approach on the multi-resolution hash encoding [58] due to its efficiency and compactness for representing and reconstructing static scenes. However, directly applying a 4D multi-resolution hash table to represent a dynamic scene would require a much larger hash table size than that for a static scene due to the much more hash collisions caused by the additional time dimension. Our finding is that, for the ubiquitous existence of static areas in a scene, storing the static points into a 4D time-dependent hash table will result in an information redundancy since each of them will occupy  $\mathcal{T}$  hash table items with an identical value for a  $\mathcal{T}$ -frame scene. It will also lead to a high hash collision rate since it narrows the capacity of the hash table, which negatively impacts the reconstruction quality. Therefore, for the points with low dynamics, we hope to establish a mechanism to reduce the frequent queries and updates to the 4D hash table and automatically save their features into a 3D hash table to avoid temporal redundancy.
From this observation, we propose Masked Space-Time Hash encoding, which combines a 3D hash encoder and a 4D hash encoder with a 3D learnable weight mask. In order to make the mask correlate with the dynamics of the corresponding positions, we adopt the Bayesian framework of Kendall and Gal [34] to estimate the uncertainty [34, 54] of each point being static. The non-linear correlation [6] between the uncertainty and the learnable mask is maximized to make the mask reflect the dynamics. In this way, the static points indicated with a low uncertainty will have a low weight for the 4D hash table and a high weight for the 3D hash table, which prevents modifications to the dynamic representations. With the proposed masked space-time hash encoding, we can set the size of a 4D hash table the same as a 3D one without much loss of rendering qualities, making the representation highly compact. Besides, without the requirements to fit the large numbers of repeated features independently, our method is easier to optimize and converge rapidly in only twenty minutes. To validate the effectiveness of our method on scenes in more realistic settings with large areas of dynamic regions and more complex movements, we collect a synchronized multi-view video dataset with 6 challenging dynamic scenes, which will be publicly available. As a result, the proposed masked space-time hash encoding achieves consistently better reconstruction metrics on two publicly available datasets consisting of 13 scenes, with only 20 minutes of training and  $130\text{MB}$  of storage. Fig. 1 and Fig. 2 show the quantitative and qualitative comparisons to other state-of-the-art methods. In summary, our contributions are:
- We propose Masked Space-time Hash Encoding, a novel representation that decomposes the dynamic radiance fields into a weighted combination of 3D and 4D hash encoders.
- The proposed method is validated on a wide range of dynamic scenes, surpassing previous state-of-the-art methods by non-trivial margins with only 20 minutes of training time and  $130\mathrm{MB}$  of memory storage.
- We propose a new synchronized multi-view dataset with more challenging dynamic scenes including scenes with many moving objects and scenes with complex and rapid movements
Neural Scene Representations for Static Scenes. Representing 3D scenes with neural networks has achieved remarkable success in recent years. NeRF [57] first proposes to use neural networks to represent radiance fields, demonstrating the remarkable potential of this representation in conjunction with volume rendering techniques. The high-fidelity rendering has sparked a surge of interest in related areas. Numerous variants have emerged to address the inherent challenges of the original approach, including improving rendering quality [3, 5, 35, 52, 90], handling sparse input [60, 13, 92, 55, 94, 17], lighting [8, 76, 62], editing [37, 48, 45, 20, 56], dealing with complicated [86, 25, 95, 36] and large scenes [81, 100, 54, 15], generalization [98, 73, 80, 12, 87, 59, 29, 88], jointly optimizing with camera poses [43, 89, 7, 30, 78, 103, 16], accelerating training [96, 14, 79, 93, 18] and speeding up rendering [71, 26, 97, 70, 47, 69, 44, 24, 28, 11, 84, 51]. Our work draws inspiration from two recent contributions in the field, namely Instant-NGP [58] and Mip-NeRF 360 [4]. In Instant-NGP, Müller et al. [58] proposes a novel data structure that leverages multi-resolution hash grids to efficiently encode the underlying voxel grid while addressing hash collisions using a decoding MLP. Furthermore, the proposed method allows for fast and memory-efficient rendering of large-scale scenes. Similarly, Mip-NeRF 360 [4] presents a sample-efficient scheme for unbound scenes, which utilizes a small density field as a sample generator and parameterizes the unbounded scene with spherical contraction. In this paper, we build upon these contributions and propose a novel approach that extends successful techniques and components to incorporate time dimension with minimal overhead. Our method is capable of representing a dynamic scene with only  $2\sim 3\times$  memory footprint than that of a static NeRF.
Novel View Synthesis for Dynamic Scenes. Extending the neural radiance field to express dynamic scenes is a natural yet challenging task that has been proven crucial to many downstream applications [39, 66, 77, 31]. Many researchers focus on monocular dynamic novel view synthesis, which takes a monocular video as input and targets to reconstruct the underlying 4D information by modeling deformation explicitly [19, 22, 41, 91, 68, 99] or implicitly [64, 23, 63, 21, 74, 49, 46, 102, 82, 32]. As an exemplary work, D-NeRF [68] models a time-varying field through a deformation network that maps a 4D coordinate into a spatial point in canonical space. Despite the great outcomes achieved by research in this line, the applicability of these methods is restricted by the inherent nature of the underlying problem [23]. A more practical way of reconstructing dynamic scenes is by employing multi-view synchronized videos [33, 50, 104, 38, 1, 75, 10, 83, 85, 2, 67]. DyNeRF [40] models dynamic scenes by exploiting a 6D plenoptic MLP with time queries and a set of difference-based importance sampling strategies. The authors also contributed to the field by presenting a real-world dataset, which validated their proposed methodology and provided a valuable resource for future research endeavors. K-Planes [72] and HexPlane [10] speed up training by decomposing the underlying 4D radiance field into several low-dimensional planes, which substantially reduces the required memory footprint and computational complexity compared with explicit 4D voxel grid. HyperReel [1] breaks down the videos into keyframe-based segments and predicts spatial offset towards the nearest keyframe. NeRFPlayer [75] and MixVoxel [83] address the problem by decomposing 4D space according to corresponding temporal properties. The former separates the original space into three kinds of fields and applies different structures and training strategies, while the latter decouples static and dynamic voxels with a variational field which further facilitates high-quality reconstruction and fast rendering. Our method implicitly decomposes 3D space with a learnable mask. This mask eliminates the requirement for manual determination of dynamic pixels and enables the acquisition of uncertainty information, facilitating high-quality reconstruction.
Our objective is to reconstruct dynamic scenes from a collection of multi-view or monocular videos with a compact representation while achieving fast training and rendering speeds. To this end, we propose Masked Space-Time Hash encoding, which represents a dynamic radiance field by a weighted combination of a 3D and a 4D hash encoder. We employ an uncertainty-guided mask learning strategy to enable the mask to learn the dynamics. In the following, we will first introduce the preliminary, then the masked space-time hash encoding and uncertainty-guided mask learning, and finally the ray sampling method.
For neural radiance fields, the input is a 3D space coordinate  $\pmb{x}$  and a direction  $\pmb{d}$  to enable the radiance field to represent the non-Lambertian effects. The output is the color  $c(\pmb{x},\pmb{d}) \in \mathbb{R}^3$  and a direction-irrelevant density  $\sigma(\pmb{x}) \in \mathbb{R}$ . Most existing methods encode the input coordinate with a mapping function  $h$  that maps the raw coordinate into Fourier features [57], voxels-based features [79, 96], hashing-based features [58] or factorized low-rank features [14]. In this work, we mainly focus on the multi-resolution hash encoding due to its efficiency and compactness. Specifically, for an input point  $\pmb{x}$ , the corresponding voxel index is computed through the scale of each level, and a hash table index is computed by a bit-wise XOR operation [61]. There are  $L$  levels of hash tables corresponding to different grid sizes in a geometric progressive manner. For each level, the feature for a continuous point is tri-linearly interpolated by the nearest grid points. The corresponding features in different levels are concatenated to obtain the final encoding. For convenience, we denote the output of the multi-resolution hash encoder for  $\pmb{x}$  as  $\mathrm{enc}(\pmb{x})$ .
After the multi-resolution hash encoder, a density-decoding MLP  $\phi_{\theta_1}$  is employed to obtain the density and an intermediate feature  $\pmb{g}$  such that:  $\sigma(\pmb{x}), \pmb{g}(\pmb{x}) = \phi_{\theta_1}(\mathbf{enc}(\pmb{x}))$ . Then, the color is computed through a color-decoding MLP  $\psi_{\theta_2} \colon c(\pmb{x}) = \psi_{\theta_2}(\pmb{g}(\pmb{x}), \pmb{d})$  which is direction-dependent.
For rendering, points along a specific ray  $\boldsymbol{r}(s) = \boldsymbol{o} + s\boldsymbol{d}$  are sampled and the volumetric rendering [57] is applied to get the rendered color  $\hat{C}(\boldsymbol{r})$ :
$$
\hat {C} (\boldsymbol {r}) = \int_ {n} ^ {f} T (s) \cdot \sigma (\boldsymbol {r} (s)) \cdot c (\boldsymbol {r} (s), \boldsymbol {d}) \mathrm {d} s, \text {w h e r e} T (s) = \exp \left(- \int_ {s _ {n}} ^ {s} \sigma (\boldsymbol {r} (s)) d s\right). \tag {1}
$$
A squared error between the rendered color  $\hat{C}(\boldsymbol{r})$  and the ground truth color  $C(\boldsymbol{r})$  is applied for back-propagation.
3.2 Masked Space-Time Hashing
For a dynamic neural radiance field, the input is a 4D space-time coordinate  $(\pmb{x},t)$ . A straightforward method is to replace the 3D hash table with a 4D one. However, this simple replacement will result in a high hash collision rate due to the enormous volume of hash queries and modifications brought by the additional time dimension. For instance, in a dynamic scene comprising  $\mathcal{T}$  frames, the hash collision rate is  $\mathcal{T}$  times higher than a static scene, leading to a degradation in the reconstruction performance. Enlarging the size of the hash table will cause an unbearable model size and be difficult to scale up with the frame numbers.
To solve this problem, we propose the masked space-time hash encoding, which incorporates a 3D multi-resolution hash mapping  $\mathrm{h}_{\mathrm{3D}}$ , a 4D multi-resolution hash mapping  $\mathrm{h}_{\mathrm{4D}}$ , and a learnable mask encoding  $\tilde{m}$ . The final encoding function is formulated as follows:
$$
\mathbf {e n c} (\boldsymbol {x}, t) = m (\boldsymbol {x}) \cdot \mathrm {h} _ {\mathrm {3 D}} (\boldsymbol {x}) + (1 - m (\boldsymbol {x})) \cdot \mathrm {h} _ {\mathrm {4 D}} (\boldsymbol {x}, t), \text {w h e r e} m (\boldsymbol {x}) = \operatorname {s i g m o i d} (\tilde {m} (\boldsymbol {x})). \tag {2}
$$
The learnable mask  $m$  can be represented by a multi-resolution hash table or a 3D voxel grid. For the 4D hash table, the sizes of time steps also adopt a geometric growing multi-resolution scheme, which is reasonable due to the natural hierarchical properties of motions in time scales.
After obtaining the space-time encoding, the density-decoding and color-decoding MLPs are applied to obtain the final outputs:
$$
\sigma (\boldsymbol {x}, t), \boldsymbol {g} (\boldsymbol {x}, t) = \phi_ {\theta_ {1}} (\operatorname {e n c} (\boldsymbol {x}, t)), \quad c (\boldsymbol {x}, \boldsymbol {d}, t) = \psi_ {\theta_ {2}} (\boldsymbol {g} (\boldsymbol {x}, t), \boldsymbol {d}). \tag {3}
$$
(a) Origin Image
(b) Mask without Uncertainty
Figure 3: We compared the learned masks  $m$  by visualizing them using volumetric rendering. As shown in the figures above, we observed that the mask learned with uncertainty is cleaner and tends to have a binarized value, which helps avoid the mixture of both tables.
(c) Mask with Uncertainty
Then the volumetric rendering is applied to each ray at each time step, and a squared error is employed as the loss function:
$$
\hat {C} (\boldsymbol {r}, t) = \int_ {n} ^ {f} T (s, t) \cdot \sigma (\boldsymbol {r} (s), t) \cdot c (\boldsymbol {r} (s), \boldsymbol {d}, t) \mathrm {d} s, \quad L _ {r} = \underset {\boldsymbol {r}, t} {\mathbb {E}} \left[ \| C (\boldsymbol {r}, t) - \hat {C} (\boldsymbol {r}, t) \| _ {2} ^ {2} \right]. \tag {4}
$$
Reduce Hash Collisions. The intuition behind the masked modeling lies in the fact that many parts in the dynamic radiance fields are time-invariant, such as the static objects, the background, etc. These static parts can be well reconstructed from the 3D hash table with a large  $m$ . For these static points, storing their features in the 4D hash table will occupy a large number of storage, largely increasing the hash collision rate. For these time-invariant parts, the 3D hash encoder can be sufficient to reconstruct, and these static parts will not modify the 4D hash table significantly when  $(1 - m(\pmb{x}))$  is small. In this way, the 4D hash table stores the properties of those points which are really dynamic, and those dynamic features in the 4D hash table will be protected by the mask term to suppress the gradients from static points.
Accelerate Rendering. Another advantage of using masked space-time hash encoding is to accelerate the fixed view-port rendering. Instead of rendering a novel view video frame-by-frame, we adopt an incremental way of using the mask to avoid redundant computations. Specifically, for a  $\mathcal{T}$ -frame scene, we first render the initial frame  $V_{0}$  as usual. The obtained mask is used to filter the static parts, and we only render the dynamic parts of other frames with the dynamic weight  $(1 - m(x)) > (1 - \epsilon)$ , where  $\epsilon$  is a hyper-parameter. In this way, except for the initial frame, we only require to render the dynamic part, improving the rendering fps from 1.4 to 15, without loss of rendering quality.
The key to the masked space-time hash encoding is that  $m(\pmb{x})$  can reflect the dynamics of the point. To this end, we design an uncertainty-guided mask learning strategy to connect the relation between the 3D hash uncertainty  $u(\pmb{x})$  and the mask  $m(\pmb{x})$ . We will elaborate on this strategy in the next part.
3.3 Uncertainty-guided Mask Learning
To make the mask  $m(\pmb{x})$  well reflect the dynamics of the corresponding point, we design an uncertainty branch in our model to estimate the uncertainty of a point being static, which is a good indicator for the dynamics of the point. To this end, the uncertainty branch is required to reconstruct the dynamic scenes only using the 3D hash table with the input-dependent uncertainties, ignoring the time input. In this way, the dynamic points are regarded as the inherent noise since its supervision are inherently inconsistent (different time step describes different geometries while the uncertainty model is time-agnostic). The inherent noise of dynamic points will lead to a high uncertainty. We adopt the Bayesian learning framework of Kendall and Gal [34] to model the heteroscedastic aleatoric uncertainty for each point.
Specifically, we construct an uncertainty field  $u$  with a voxel grid representation.  $u(\pmb{x})$  denotes the uncertainty of a space point  $\pmb{x}$ . We denote the raw output of the uncertainty voxel-grid as  $\tilde{u}$ , and a soft-plus is used as the activation:  $u(\pmb{x}) = u_m + \log(1 + \exp(\tilde{u}(\pmb{x}))$ , where  $u_m$  is a hyper-parameter for shifting the uncertainty values [54]. For each ray, the ray-level uncertainty  $U(\pmb{r})$  is calculated through volumetric rendering [53]:
$$
\mathrm {U} (\boldsymbol {r}) = \int_ {n} ^ {f} T (s) \cdot \sigma (\boldsymbol {r} (s)) \cdot u (\boldsymbol {r} (s)) \mathrm {d} s. \tag {5}
$$
Besides, the color and density estimated by the 3D hash table are:
$$
\sigma_ {s} (\boldsymbol {x}), \boldsymbol {g} _ {s} (\boldsymbol {x}) = \phi_ {\theta_ {1}} \left(\mathrm {h} _ {\mathrm {3 D}} (\boldsymbol {x})\right), \quad c _ {s} (\boldsymbol {x}, \boldsymbol {d}) = \psi_ {\theta_ {2}} \left(\boldsymbol {g} _ {s} (\boldsymbol {x}), \boldsymbol {d}\right). \tag {6}
$$
The rendered color of this branch is  $\hat{C}_s(r)$  by applying the volumetric rendering. After that, the uncertainty-based loss for ray  $r$  is defined as:
$$
L _ {u} = \underset {\boldsymbol {r}, t} {\mathbb {E}} \left[ \frac {1}{2 \mathrm {U} (\boldsymbol {r}) ^ {2}} \| C (\boldsymbol {r}, t) - \hat {C} _ {s} (\boldsymbol {r}) \| _ {2} ^ {2} + \log \mathrm {U} (\boldsymbol {r}) \right]. \tag {7}
$$
Note that the uncertainty branch is time-agnostic, i.e., the predicted color  $\hat{C}(\boldsymbol{r})$  is not time-dependent, which is important to make the uncertainty relevant to dynamics. In this way, the uncertainty for each point  $u(\boldsymbol{x})$  can be estimated through the above loss function. For the dynamic points, the uncertainty is inherently large because of the ambiguous and inconsistent geometries caused by the inconsistent pixel color supervision. In this perspective, the uncertainty can well reflect the dynamics of each point and could guide the mask values.
Bridging uncertainty with mask. Though we find the correlation between the mask  $m$  and uncertainty  $u$ , it is not trivial to connect them. First,  $m$  and  $u$  are in different value ranges,  $m(\pmb{x}) \in [0,1]$  while  $u(\pmb{x}) \in [0, +\infty)$ . Second, the distributions of  $m$  and  $u$  are very different, and the relations between them are non-linear. Imposing a hard relationship between them will impact the training of their specific branches. We instead maximize the mutual information between the two random variables  $m$  and  $u$  to maximize the nonlinear correlation between them. Although MI can not measure whether the correlation is negative, Eq. (7) can guarantee this. The mutual information  $I(m,u)$  describes the decrease of uncertainty in  $m$  given  $u$ :  $I(m,u) := H(m) - H(m|u)$ , where  $H$  is the Shannon entropy. To estimate the mutual information between  $m$  and  $u$ , we adopt the neural estimator in [6] to approximate the mutual information  $I(m,u)$  as:
$$
I _ {\Theta} (m, u) = \sup  _ {\theta \in \Theta} \mathbb {E} _ {\mathbb {P} _ {m, u}} \left[ T _ {\theta} \right] - \log \left(\mathbb {E} _ {\mathbb {P} _ {m} \otimes \mathbb {P} _ {u}} \left[ e ^ {T _ {\theta}} \right]\right). \tag {8}
$$
$\mathbb{P}_{m,u}$  is the joint probability distribution of  $m$  and  $u$ , and  $\mathbb{P}_m\otimes \mathbb{P}_u$  is the product of marginals.  $T_{\theta}$  is a neural network with parameters  $\theta \in \Theta$ . We choose a small MLP with two hidden layers to represent  $T_{\theta}$  and draw samples from the joint distribution and the product marginals to compute the empirical estimation of the expectation. By maximizing the estimated mutual information, we build the correlation between  $m$  and  $u$ . At last, the overall learning objective of MSTH is the combination of the above losses:
$$
L = L _ {r} + \lambda \cdot L _ {u} - \gamma \cdot I _ {\Theta} (m, u), \tag {9}
$$
where  $\lambda$  and  $\gamma$  are two hyper-parameters. For rendering a novel view, the uncertainty branches and the mutual information network are disabled.