SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Pengchong Hu Zhizhong Han
Machine Perception Lab, Wayne State University, Detroit, USA
pchu@wayne.edu h312h@wayne.edu
Learning neural implicit representations has achieved remarkable performance in 3D reconstruction from multi-view images. Current methods use volume rendering to render implicit representations into either RGB or depth images that are supervised by multi-view ground truth. However, rendering a view each time suffers from incomplete depth at holes and unawareness of occluded structures from the depth supervision, which severely affects the accuracy of geometry inference via volume rendering. To resolve this issue, we propose to learn neural implicit representations from multi-view RGBD images through volume rendering with an attentive depth fusion prior. Our prior allows neural networks to perceive coarse 3D structures from the Truncated Signed Distance Function (TSDF) fused from all depth images available for rendering. The TSDF enables accessing the missing depth at holes on one depth image and the occluded parts that are invisible from the current view. By introducing a novel attention mechanism, we allow neural networks to directly use the depth fusion prior with the inferred occupancy as the learned implicit function. Our attention mechanism works with either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene in the context of Simultaneous Localization and Mapping (SLAM). Our evaluations on widely used benchmarks including synthetic and real-world scans show our superiority over the latest neural implicit methods. Please see our project page for code and data at https://machineperceptionlab.github.io/Attentive_DF_Prior/.
3D reconstruction from multi-view images has been studied for decades. Traditional methods like Structure from Motion (SfM) [61], Multi-View Stereo (MVS) [62], and Simultaneous Localization and Mapping (SLAM) [49] estimate 3D structures as point clouds by maximizing multi-view color consistency. Current methods [84, 28] mainly adopt data-driven strategies to learn depth estimation priors from large scale benchmarks using deep learning models. However, the reconstructed 3D point clouds lack geometry details, due to their discrete representation essence, which makes them not friendly to downstream applications.
More recent methods use implicit functions such as signed distance functions (SDFs) [77] or occupancy functions [51] as continuous representations of 3D shapes and scenes. Using volume rendering, we can learn neural implicit functions by comparing their 2D renderings with multi-view ground truth including color [88, 97], depth [88, 3, 97] or normal [88, 76, 16] maps. Although the supervision of using depth images as rendering target can provide detailed structure information and guide importance sampling along rays [88], both the missing depth at holes and the unawareness of occluded structures make it hard to significantly improve the reconstruction accuracy. Hence, how to more
effectively leverage depth supervision for geometry inference through volume rendering is still a challenge.
To overcome this challenge, we introduce to learn neural implicit through volume rendering with an attentive depth fusion prior. Our key idea is to provide neural networks the flexibility of choosing geometric clues, i.e., the geometry that has been learned and the Truncated Signed Distance Function (TSDF) fused from all available depth images, and combining them into neural implicit for volume rendering. We regard the TSDF as a prior sense of the scene, and enable neural networks to directly use it as a more accurate representation. The TSDF enables accessing the missing depth at holes on one depth image and the occluded structures that are invisible from the current view, which remedies the demerits of using depth ground truth to supervise rendering. To achieve this, we introduce an attention mechanism to allow neural networks to balance the contributions of currently learned geometry and the TSDF in the neural implicit, which leads the TSDF into an attentive depth fusion prior. Our method works with either known camera poses or camera tracking in the context of SLAM, where our prior could be either a one-time fused TSDF that represents a whole scene or an incrementally fused TSDF that represents a partial scene. We evaluate our performance on benchmarks containing synthetic and real-world scans, and report our superiority over the latest methods with known or estimated camera poses. Our contributions are listed below.
i) We present a novel volume rendering framework to learn neural implicit representations from RGBD images. We enable neural networks to use either currently learned geometry or the one from depth fusion in volume rendering, which leads to a novel attentive depth fusion prior for learning neural implicit functions inheriting the merits of both the TSDF and the inference.
ii) We introduce a novel attention mechanism and a neural network architecture to learn attention weights for the attentive depth fusion prior in neural rendering with either known camera poses or camera pose tracking in SLAM.
iii) We report the state-of-the-art performance in surface reconstruction and camera tracking on benchmarks containing synthetic and real-world scans.
Learning 3D implicit functions using neural networks has made huge progress [61, 62, 46, 51, 88, 78, 73, 76, 16, 56, 34, 25, 11, 93, 42, 44, 9, 43, 4, 33, 5, 19, 26, 47, 20, 70, 12, 48, 55, 17, 27, 79, 35, 32, 21, 7, 58, 86, 52]. We can learn neural implicit functions from 3D ground truth [23, 8, 54, 45, 68, 39, 69], 3D point clouds [94, 41, 15, 1, 92, 2, 10] or multi-view images [46, 14, 51, 77, 88, 78, 73, 76, 16]. We briefly review methods with multi-view supervision below.
2.1 Multi-view Stereo
Classic multi-view stereo (MVS) [61, 62] employs multi-view photo consistency to estimate depth maps from multiple RGB images. They rely on matching key points on different views, and are limited by large viewpoint variations and complex illumination. Without color, space carving [30] is also an effective way of reconstructing 3D structure as voxel grids.
Recent methods employ data driven strategy to train neural networks to predict depth maps from either depth supervision [84] or multi-view photo consistency [95].
2.2 Neural Implicit from Multi-view Supervision
Early works leverage various differentiable renderers [63, 38, 26, 89, 37, 80, 50, 36, 67] to render the learned implicit functions into images, so that we can measure the error between rendered images and ground truth images. These methods usually require masks to highlight the object, and use surface rendering to infer the geometry, which limits their applications in scenes. Similarly, DVR [50] and IDR [87] predict the radiance near surfaces.
With volume rendering, NeRF [46] and its variations [53, 47, 57, 60, 97, 3, 75, 6, 98, 67] model geometry and color together. They can generate plausible images from novel viewpoints, and do not need masks during rendering procedure. UNISURF [51] and NeuS [77] learn occupancy functions and SDFs by rendering them with colors using revised rendering equations. Following methods
Figure 1: Overview of our method.
improve accuracy of implicit functions using additional priors or losses related to depth [88, 3, 97], normals [88, 76, 16], and multi-view consistency [14].
Depth images are also helpful to improve the inference accuracy. Depth information can guide sampling along rays [88] or provide rendering supervision [88, 3, 97, 96, 24, 31, 82, 98], which helps neural networks to estimate surfaces.
2.3 Neural Implicit with SLAM
Given RGBD images, more recent methods [91, 81, 72, 59] employ neural implicit representations in SLAM. iMAP [66] shows that an MLP can serve as the only scene representation in a realtime SLAM system. NICE-SLAM [97] introduces a hierarchical scene representation to reconstruct large scenes with more details. NICER-SLAM [96] uses easy-to-obtain monocular geometric cues without requiring depth supervision. Co-SLAM [74] jointly uses coordinate and sparse parametric encodings to learn neural implicit functions. Segmentation priors [29, 18] show their potentials to improve the performance of SLAM systems. With segmentation priors, vMAP [29] represents each object in the scene as a neural implicit in a SLAM system.
Instead of using depth ground truth to only supervise rendering, which contains incomplete depth and unawareness of occlusion, we allow our neural network to directly use depth fusion priors as a part of neural implicit, and determine where and how to use depth priors along with learned geometry.
Overview. We present an overview of our method in Fig. 1. Given RGBD images  $\{I_j, D_j\}_{j=1}^J$  from  $J$  view angles, where  $I_j$  and  $D_j$  denote the RGB and depth images, available either all together or in a streaming way, we aim to infer the geometry of the scene as an occupancy function  $f$  which predicts the occupancy  $f(\mathbf{q})$  at arbitrary locations  $\mathbf{q} = (x, y, z)$ . Our method works with camera poses  $\{M_j\}$  that are either known or estimated by our camera tracking method in the context of SLAM.
We learn the occupancy function through volume rendering using the RGBD images  $\{I_j, D_j\}_{j=1}^J$  as supervision. We render  $f$  with a color function  $c$  which predicts an RGB color  $c(\pmb{q})$  at locations  $\pmb{q}$  into an RGB image  $I_j'$  and a depth image  $D_j'$ , which are optimized to minimize their rendering errors to the supervision  $\{I_j, D_j\}$ .
We start from shooting rays  $\{V_k\}$  from current view  $I_j$ , and sample queries  $\mathbf{q}$  along each ray  $V_k$ . For each query  $\mathbf{q}$ , we employ learnable feature grids  $G_l$ ,  $G_h$ , and  $G_c$  covering the scene to interpolate its hierarchical geometry features  $\pmb{t}_l$ ,  $\pmb{t}_h$ , and a color feature  $\pmb{t}_c$  by trilinear interpolation. Each feature is further transformed into an occupancy probability or RGB color by their corresponding decoders  $f_l$ ,  $f_h$ , and  $f_c$ . For queries  $\mathbf{q}$  inside the bandwidth of a TSDF grid  $G_s$  fused from available depth images  $\{D_i\}$ , we leverage its interpolation from  $G_s$  as a prior of coarse occupancy estimation. The prior is attentive by a neural function  $f_a$  which determines the occupancy function  $f(\mathbf{q})$  by combining currently learned geometry and the coarse estimation using the learned attention weights from  $f_a$ .
Lastly, we use the occupancy function  $f(\pmb{q})$  and the color function  $c(\pmb{q})$  to render  $I'$  and  $D'$  through volume rendering. With a learned  $f$ , we run the marching cubes [40] to reconstruct a surface.
Depth Fusion and Bandwidth Awareness. With known or estimated camera poses  $\{M_j\}$ , we get a TSDF by fusing depth images  $\{D_j\}$  that are available either all together or in a streaming way. Our method can work with a TSDF that describes either a whole scene or just a part of the scene, and we will report the performance with different settings in ablation studies. We use the TSDF to provide a coarse occupancy estimation which removes the limit of the missing depth or the unawareness of occlusion on single depth supervision. The TSDF is a grid which predicts signed distances at any locations inside it through trilinear interpolation. The predicted signed distances are truncated with a threshold  $\mu$  into a range of  $[-1, 1]$  which covers a band area on both sides of the surface in Fig. 1.
Since the TSDF predicts signed distances for queries within the bandwidth with higher confidence than the ones outside the bandwidth, we only use the depth fusion prior within the bandwidth. This leads to different ways of learning geometries, as shown in Fig. 1. Specifically, for queries  $\mathbf{q}$  within the bandwidth, we model the low-frequency surfaces using a low resolution feature grid  $G_{l}$ , learn the high-frequency details as a complementation using a high resolution feature grid  $G_{h}$ , and let our attention mechanism to determine how to use the depth fusion prior from the TSDF  $G_{s}$ . Instead, we only use the low resolution feature grid  $G_{l}$  to model densities outside the bandwidth during volume rendering. Regarding color modeling, we use the feature grid  $G_{c}$  with the same size for interpolating color of queries over the scene.
Feature Interpolation. For queries  $\mathbf{q}$  sampled along rays, we use the trilinear interpolation to obtain its features  $\mathbf{h}_l$ ,  $\mathbf{h}_h$ , and  $\mathbf{h}_c$  from learnable  $G_l$ ,  $G_h$ ,  $G_c$  and occupancy estimation  $o_s$  from  $G_s$ . Both feature grids and the TSDF grid cover the whole scene and use different resolutions, where learnable feature vectors are associated with vertices on each feature grid. Signed distances at vertices on  $G_s$  may change if we incrementally fuse depth images in the context of SLAM.
Occupancy Prediction Priors. Similar to NICE-SLAM [97], we use pre-trained decoders  $f_{l}$  and  $f_{h}$  to predict low frequency occupancies  $o_{l}$  and high frequency ones  $o_{h}$  from the interpolated features  $t_{l}$  and  $t_{h}$ , respectively. We use  $f_{l}$  and  $f_{h}$  as an MLP decoder in ConvONet [54] respectively, and minimize the binary cross-entropy loss to fit the ground truth. After pre-training, we fix the parameters in  $f_{l}$  and  $f_{h}$ , and use them to predict occupancies below,
$$
o _ {l} = f _ {l} (\boldsymbol {q}, \boldsymbol {t} _ {l}), o _ {h} = f _ {h} (\boldsymbol {q}, \boldsymbol {t} _ {h}), o _ {l h} = o _ {l} + o _ {h}, \tag {1}
$$
where we enhance  $f_{h}$  by concatenating  $f_{l}$  as  $f_{h} \gets [f_{h}, f_{l}]$  and denote  $o_{lh}$  as the occupancy predicted at query  $q$  by the learned geometry.
Color Predictions. Similarly, we use the interpolated feature  $t_c$  and an MLP decoder  $f_c$  to predict color at query  $q$ , i.e.,  $c(q) = f_c(q, t_c)$ . We predict color to render RGB images for geometry inference or camera tracking in SLAM. The decoder is parameterized by parameters  $\theta$  which are optimized with other learnable parameters in the feature grids.
Attentive Depth Fusion Prior. We introduce an attention mechanism to leverage the depth fusion prior. We use a deep neural network to learn attention weights to allow networks to determine how to use the prior in volume rendering.
As shown in Fig. 1, we interpolate a signed distance  $s \in [-1,1]$  at query  $\mathbf{q}$  from the TSDF  $G_{s}$  using trilinear interpolation, where  $G_{s}$  is fused from available depth images. We formulate this interpolation as  $s = f_{s}(\mathbf{q}) \in [-1,1]$ . We normalize  $s$  into an occupancy  $o_{s} \in [0,1]$ , and regard  $o_{s}$  as the occupancy predicted at query  $\mathbf{q}$  by the depth fusion prior.
For query  $\mathbf{q}$  within the bandwidth, our attention mechanism trains an MLP  $f_{a}$  to learn attention weights  $\alpha$  and  $\beta$  to aggregate the occupancy  $o_{lh}$  predicted by the learned geometry and the occupancy  $o_{s}$  predicted by the depth fusion prior, which leads the TSDF to an attentive depth fusion prior. Hence, our attention mechanism can be formulated as,
$$
[ \alpha , \beta ] = f _ {a} \left(o _ {l h}, o _ {s}\right), \text {a n d} \alpha + \beta = 1. \tag {2}
$$
We implement  $f_{a}$  using an MLP with 6 layers. The reason why we do not use a complex network like Transformer is that we want to justify the effectiveness of our idea without taking too much credit from complex neural networks. Regarding the design, we do not use coordinates or positional encoding as a part of input to avoid noisy artifacts. Moreover, we leverage a Softmax normalization layer to achieve  $\alpha + \beta = 1$ , and we do not predict only one parameter  $\alpha$  and use  $1 - \alpha$  as the second weight, which also degenerates the performance. We will justify these alternatives in experiments.
For query  $\pmb{q}$  outside the bandwidth, we predict the occupancy using the feature  $t_l$  interpolated from the low frequency grid  $G_l$  and the decoder  $f_l$  to describe the relatively simpler geometry. In summary, we eventually formulate our occupancy function  $f$  as a piecewise function below,
$$
f (\boldsymbol {q}) = \left\{ \begin{array}{l l} \alpha \times o _ {l h} + \beta \times o _ {s}, & f _ {s} (\boldsymbol {q}) \in (- 1, 1) \\ o _ {l}, & f _ {s} (\boldsymbol {q}) = 1 \text {o r} - 1 \end{array} \right. \tag {3}
$$
Volume Rendering. We render the color function  $c$  and occupancy function  $f$  into RGB  $I'$  and depth  $D'$  images to compare with the RGBD supervision  $\{I, D\}$ .
With camera poses  $M_{j}$ , we shoot a ray  $V_{k}$  from view  $I_{j}$ .  $V_{k}$  starts from the camera origin  $m$  and points a direction of  $r$ . We sample  $N$  points along the ray  $V_{k}$  using stratified sampling and uniformly sampling near the depth, where each point is sampled at  $q_{n} = m + d_{n}r$  and  $d_{n}$  corresponds to the depth value of  $q_{n}$  on the ray. Following UNISURF [51], we transform occupancies  $f(q_{n})$  into weights  $w_{n}$  which is used for color and depth accumulation along the ray  $V_{k}$  in volume rendering,
$$
w _ {n} = f \left(\boldsymbol {q} _ {n}\right) \prod_ {n ^ {\prime} = 1} ^ {n - 1} \left(1 - f \left(\boldsymbol {q} _ {n ^ {\prime}}\right)\right), I (k) ^ {\prime} = \sum_ {n ^ {\prime} = 1} ^ {N} w _ {n ^ {\prime}} \times c \left(\boldsymbol {q} _ {n ^ {\prime}}\right), D (k) ^ {\prime} = \sum_ {n ^ {\prime} = 1} ^ {N} w _ {n ^ {\prime}} \times d _ {n ^ {\prime}}. \tag {4}
$$
Loss. With known camera pose  $M_{j}$ , we render the scene into the color and depth images at randomly sampled  $K$  pixels on the  $j$ -th view, and optimize parameters by minimizing the rendering errors,
$$
L _ {I} = \frac {1}{J K} \sum_ {j, k = 1} ^ {J, K} \left\| I _ {j} (k) - I _ {j} ^ {\prime} (k) \right\| _ {1}, L _ {D} = \frac {1}{J K} \sum_ {j, k = 1} ^ {J, K} \left\| D _ {j} (k) - D _ {j} ^ {\prime} (k) \right\| _ {1}, \min  _ {\boldsymbol {\theta}, G _ {l}, G _ {h}, G _ {c}} L _ {D} + \lambda L _ {I}. \tag {5}
$$
In the Context of SLAM. We can jointly do camera tracking and learning neural implicit from streaming RGBD images. To achieve this, we regard the camera extrinsic matrix  $M_{j}$  as learnable parameters, and optimize them by minimizing our rendering errors. Here, we follow [97] to weight the depth rendering loss to decrease the importance at pixels with large depth variance along the ray,
$$
\min  _ {M _ {j}} \frac {1}{J K} \sum_ {j, k = 1} ^ {J, K} \frac {1}{\operatorname {V a r} \left(D _ {j} ^ {\prime} (k)\right)} \left| \left| D _ {j} (k) - D _ {j} ^ {\prime} (k) \right| \right| _ {1} + \lambda_ {1} L _ {I}, \tag {6}
$$
where  $Var(D_j'(k)) = \sum_{n=1}^{N} w_n (D_j'(k) - d_n)^2$ . Moreover, with streaming RGBD images, we incrementally fuse the most current depth image into TSDF  $G_s$ . Specifically, the incremental fusion includes a pre-fusion and an after-fusion stage. The pre-fusion aims to use a camera pose coarsely estimated by a traditional method to fuse a depth image onto a current TSDF to calculate a rendering error for a more accurate pose estimation at current frame. The after-fusion stage will refuse the depth image onto the current TSDF for camera tracking at the next frame. Please refer to our supplementary materials for more details.
We do tracking and mapping iteratively. For mapping procedure, we render  $E$  frames each time and back propagate rendering errors to update parameters.  $E$  frames include the current frame and  $E - 1$  key frames that have overlaps with the current frame. For simplicity, we merely maintain a key frame list by adding incoming frames with an interval of 50 frames for fair comparisons.
Details. The optimization is performed at three stages, which makes optimization converge better. We first minimize  $L_{D}$  by optimizing low frequency feature grid  $G_{l}$ , and then both low and high frequency feature grid  $G_{l}$  and  $G_{h}$ , and finally minimize Eq 5 by optimizing  $G_{l}$ ,  $G_{h}$ , and  $G_{c}$ . Our bandwidth from the TSDF  $G_{s}$  covers 5 voxels on both sides of the surface. We shoot  $K = 1000$  or 5000 rays for reconstruction or tracking from each view, and render  $E = 5$  or 10 frames each time for fair comparison with other methods. We set  $\lambda = 0.2$ ,  $\lambda_{1} = 0.5$  in loss functions. We sample  $N = 48$  points along each ray for rendering. More implementation details can be found in our supplementary materials.
Datasets. We report evaluations on both synthetic datasets and real scans including Replica [64] and ScanNet [13]. For fair comparisons, we report results on the same scenes from Replica and ScanNet
Table 1: Reconstruction Comparisons on Replica.
COLMAP [62]	TSDF [90]	iMAP [66]	DI [22]	NICE [97]	Vox [83]	DROID [71]	NICER [96]	Ours	vMAP [29]	Ours*
DepthL1 ↓	-	6.56	7.64	23.33	3.53	-	-	-	3.01	-	2.60
Acc. ↓	8.69	1.56	6.95	19.40	2.85	2.67	5.50	3.65	2.77	3.20	2.59
Comp. ↓	12.12	3.33	5.33	10.19	3.00	4.55	12.29	4.16	2.45	2.39	2.28
Ratio ↑	67.62	87.61	66.60	72.96	89.33	86.59	63.62	79.37	92.79	92.99	93.38
as the latest methods. Specifically, we report comparisons on all 8 scenes in Replica. As for ScanNet, we report comparison on scene 50, 84, 580, and 616 used by MonoSDF [88], and also scene 59, 106, 169, and 207 used by NICE-SLAM [97]. We mainly report average results over the dataset, please refer to our supplementary materials for results on each scene.
Metrics. With the learned occupancy function  $f$ , we reconstruct the surface of a scene by extracting the zero level set of  $f$  using the marching cubes algorithm [40]. Following previous studies [97, 22], we use depth L1 [cm], accuracy [cm], completion [cm], and completion ratio  $[<5\mathrm{cm}\%]$  as metrics to evaluate reconstruction accuracy on Replica. Additionally, we report Chamfer distance (L1), precision, recall, and F-score with a threshold of 0.05 to evaluate reconstruction accuracy on ScanNet. To evaluate the accuracy in camera tracking, we use ATE RMSE [65] as a metric. We follow the same parameter setting in these metrics as [97].
Evaluations on Replica. We use the same set of RGBD images as [66, 96]. We report evaluations in surface reconstruction and camera tracking in Tab. 1, Tab. 2 and Tab. 3, respectively.
We jointly optimize camera poses and learn geometry in the context of SLAM. The depth fusion prior  $G_{s}$  incrementally fuses depth images using estimated camera poses. We report the accuracy of reconstruction from  $G_{s}$  as "TSDF". Compared to this baseline, we can see that our method improves the reconstruction using the geometry learned through volume rendering and occupancy prediction priors. We visualize the advantage of learning geometry in Fig. 2. Note that the holes in TSDF are caused by the absence of RGBD scans. We can see that our neural implicit keeps the correct structure in TSDF and plausibly completes the missing structures in TSDF. We further visualize the attention weights  $\beta$  learned for depth fusion in Fig. 3. We visualize the cross sections on 4 scenes, where  $\beta$  learned in bandwidth are shown in color red. Generally, the neural implicit mostly focuses
Figure 2: Merits of attentive depth fusion prior.
more on the depth fusion priors in areas where TSDF is complete, while focusing more on the learned geometry in areas where TSDF is incomplete. In the area where TSDF is complete, the network also pays some attention to the inferred occupancy because the occupancy interpolated from TSDF may not be accurate, especially on the most front of surfaces in Fig. 3.
Moreover, our method outperforms the latest implicit-based SLAM methods like NICE-SLAM [97] and NICER-SLAM [96]. We present visual comparisons in Fig. 6, where our method produces more accurate and compact geometry. For method using GT camera poses like vMAP [29] and MonoSDF [88], we achieve much better performance as shown by "Ours*" in Tab. 1 and "Ours" in Tab. 2. Although we require the absolute dense depth maps, compared to MonoSDF [88] that can work with
Figure 3: Visualization of attention on depth fusion.