SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Reconstruction Comparison with MonoSDF on Replica.
Test Split	Train Split
Normal C.↑	Chamfer-L1↓	F-score ↑	Normal C.↑	Chamfer-L1↓	F-score ↑
MonoSDF [88]	90.56	4.26	76.42	91.80	3.59	85.67
Ours	90.69	2.43	92.47	91.05	2.73	90.52
Table 3: Camera Tracking Comparisons (ATE RMSE) on Replica.
rm-0	rm-1	rm-2	off-0	off-1	off-2	off-3	off-4	Avg.
NICE-SLAM [97]	1.69	2.04	1.55	0.99	0.90	1.39	3.97	3.08	1.95
NICER-SLAM [96]	1.36	1.60	1.14	2.12	3.23	2.12	1.42	2.01	1.88
Ours	1.39	1.55	2.60	1.09	1.23	1.61	3.61	1.42	1.81
scaled depth maps, our method only can use the views in front of the current time step, where MonoSDF [88] can utilize all views during the whole training process.
In camera tracking, our results in Tab. 3 achieve the best in average. We compare the estimated trajectories on scene off-4 in Fig. 4 (c). The comparison shows that our attentive depth fusion prior can also improve the camera tracking performance through volume rendering.
Evaluations on ScanNet. We further evaluate our method on ScanNet. For surface reconstruction, we compare with the latest methods for learning
Figure 4: Visual comparisons in camera tracking.
neural implicit from multiview images. We report both their results and ours with GT camera poses, where we also fuse every 10 depth images into the TSDF  $G_{s}$  and render every 10 frames for fair comparisons. Numerical comparisons in Tab. 4 show that our method achieves the best performance in terms of all metrics, where we use the culling strategy introduced in MonoSDF [88] to clean the reconstructed mesh. We highlight our significant improvements in visual comparisons in Fig. 7. We see that our method can reconstruct sharper, more compact and detailed surfaces than other methods. We detail our comparisons on every scene with the top results reported by GO-Surf [75] and NICE-SLAM [97] in Tab. 5 and Tab. 6. The comparisons in Tab. 5 show that our method achieves higher reconstruction accuracy while GO-Surf produces more complete surfaces. We present visual comparisons with error maps in Fig. 5.
In camera tracking, we compare with the latest methods. We incrementally fuse the most current depth frames into TSDF  $G_{s}$  which is used for attentive depth fusion priors in volume rendering. Numerical comparisons in Tab. 7 show that our results are better on 2 out of 4 scenes and achieve the best in average. Tracking trajectory comparisons in Fig. 4 (a) and (b) also show our superiority.
Figure 5: Visual comparison of Error maps (Red: Large).
Figure 6: Visual comparisons of error maps (Red: Large) in surface reconstructions on Replica.
Figure 7: Visual comparisons in surface reconstructions on ScanNet.
Table 4: Reconstruction Comparisons on ScanNet.
Acc ↓	Comp ↓	Chamfer-L1 ↓	Prec ↑	Recall ↑	F-score ↑
COLMAP [62]	0.047	0.235	0.141	0.711	0.441	0.537
UNISURF [51]	0.554	0.164	0.359	0.212	0.362	0.267
NeuS [77]	0.179	0.208	0.194	0.313	0.275	0.291
VolSDF [85]	0.414	0.120	0.267	0.321	0.394	0.346
Manhattan-SDF [16]	0.072	0.068	0.070	0.621	0.586	0.602
NeuRIS [76]	0.050	0.049	0.050	0.717	0.669	0.692
MonoSDF [88]	0.035	0.048	0.042	0.799	0.681	0.733
Ours	0.034	0.039	0.037	0.913	0.894	0.902
Table 5: Reconstruction Comparison with GO-Surf on ScanNet.
Scene ID	GO-Surf [75]	Ours
0050	0084	0580	0616	Avg.	0050	0084	0580	0616	Avg.
Acc ↓	0.056	0.073	0.057	0.026	0.053	0.030	0.039	0.041	0.026	0.034
Comp ↓	0.024	0.017	0.024	0.023	0.022	0.043	0.014	0.035	0.063	0.039
Chamfer-L1 ↓	0.040	0.045	0.040	0.025	0.038	0.037	0.026	0.038	0.045	0.037
Table 6: Reconstruction Comparison with NICE-SLAM on ScanNet.
Scene ID	NICE [97]	Ours
0050	0084	0580	0616	Avg.	0050	0084	0580	0616	Avg.
Acc ↓	0.030	0.031	0.032	0.026	0.030	0.030	0.039	0.041	0.026	0.034
Comp ↓	0.053	0.020	0.031	0.076	0.045	0.043	0.014	0.035	0.063	0.039
Chamfer-L1 ↓	0.041	0.025	0.032	0.051	0.037	0.037	0.026	0.038	0.045	0.037
We report ablation studies to justify the effectiveness of modules in our method on Replica. We use estimated camera poses in ablation studies.
Effect of Depth Fusion. We first explore the effect of different ways of depth fusion on the performance. According to whether we use GT camera poses or incremental fusion, we try 4 alternatives with  $G_{s}$  obtained by fusing all depth images at the very beginning using GT camera poses (Offline) or fusing the current depth image incrementally (Online), and using tracking to estimate camera poses (Tracking) or GT camera poses (GT) in Tab. 8. The comparisons show that our method can work well with either GT or estimated camera poses and fusing depth either all together or incrementally in a streaming way. Additional conclusion includes that GT camera poses in either depth fusion and rendering do improve the reconstruction accuracy, and the structure fused from more recent frames is more important than the whole structure fused from all depth images.
Table 7: Camera Tracking Comparisons (ATE RMSE) on ScanNet.
Scene ID	0059	0106	0169	0207	Avg.
iMAP [66]	32.06	17.50	70.51	11.91	33.00
DI [22]	128.00	18.50	75.80	100.19	80.62
NICE [97]	12.25	8.09	10.28	5.59	9.05
CO [74]	12.29	9.57	6.62	7.13	8.90
Ours	10.50	7.48	9.31	5.67	8.24
Table 8: Ablation Studies on Depth Fusion.
TSDF	Offline+GT	Online+GT	Offline+Tracking	Online+Tracking (Ours)
DepthL1 ↓	6.56	2.79	2.73	2.75	3.01
Acc. ↓	1.56	2.62	2.62	2.82	2.77
Comp. ↓	3.33	2.45	2.36	2.40	2.45
Ratio ↑	87.61	92.78	92.79	92.97	92.79
Effect of Attention. We explore the effect of attention on the performance in Tab. 9. First of all, we remove the attention mechanism, and observe a severe degeneration in accuracy, which indicates that the attention plays a big role in directly applying depth fusion priors in
neural implicit. Then, we try to apply attention of depth fusion priors on geometries predicted by different features including low frequency  $t_l$ , high frequency  $t_h$ , and both of them. The comparisons show that adding details from depth fusion priors onto low frequency surfaces does not improve the performance. More analysis on attention mechanism can be found in Fig. 9.
Attention Alternatives. Our preliminary results show that using softmax normalization layer achieves better performance than other alternatives. Since we merely need two attention weights  $\alpha$  and  $\beta$  to combine the learned geometry and the depth fusion prior, one alternative is to use a sigmoid function to predict
one attention weight and use its difference to 1 as another attention weight. However, sigmoid can not effectively take advantage of the depth fusion prior. We also try to use coordinates as an input, which pushes the network to learn spatial sensitive attentions. While the reconstructed surfaces turn out to be noisy in Fig. 8. This indicates that our attention learned a general attentive pattern for all locations in the scene. We also report the numerical comparisons in Tab. 10. Please refer to our supplementary materials for more results related to attention alternatives.
Table 9: Ablation Studies on Attention.
w/o Attention	Low	Low + High	High (Ours)
DepthL1 ↓	1.86	2.75	2.96	1.44
Acc. ↓	2.69	2.96	3.85	2.54
Comp. ↓	2.81	3.14	2.91	2.41
Ratio ↑	91.46	89.73	93.63	93.22
Figure 8: Visual comparison of error maps with different attention alternatives (Red: Large).
Effect of Bandwidth. We further study the effect of bandwidth on the performance in Tab. 11. We try to use attentive depth fusion priors everywhere in the scene with no bandwidth. The degenerated results indicate that the truncated area outside the bandwidth does not provide useful structures to improve the performance. Moreover, wider bandwidth may cause more artifacts caused by the calculation of the TSDF  $G_{s}$  while narrower bandwidth brings more incomplete structures, neither of which improves the performance. Instead of using low frequency geometry outside the bandwidth, we also try to use high frequency surfaces. We can see that low frequency geometry is more suitable to describe the scene outside the bandwidth.
Table 11: Ablation Studies on the Effect of Bandwidth.
w/o Bandwidth	Bandwidth=3	Bandwidth=5 (Ours)	Bandwidth=7	High	Low (Ours)
DepthL1 ↓	1.87	1.61	1.44	2.01	124.21	1.44
Acc. ↓	2.40	2.85	2.54	2.93	25.06	2.54
Comp. ↓	3.49	2.61	2.41	2.73	3.62	2.41
Ratio ↑	90.94	92.84	93.22	93.55	86.92	93.22
Effect of Prediction Priors. Prediction priors from decoders  $f_{l}$  and  $f_{h}$  are also important for accuracy improvements. Instead of using fixed parameters in these decoders, we try to train  $f_{l}$  (Fix  $f_{h}$ ) or  $f_{h}$  (Fix  $f_{l}$ ) with other parameters. Numerical comparisons in Tab. 12 show that
Table 12: Ablation Studies on Prediction Priors.
w/o Fix	Fix fl	Fix fh	Fix fl +fh (Ours)
DepthL1 ↓	125.95	43.46	F	1.44
Acc. ↓	127.62	67.83	F	2.54
Comp. ↓	56.73	18.79	F	2.41
Ratio ↑	5.27	103.51	F	93.22
training  $f_{l}$  or  $f_{h}$  cannot improve the performance, or even fails in camera tracking (Fix  $f_{h}$ ).
More Analysis on How Attention Works. Additionally, we do a visual analysis on how attention works in Fig. 9. We sample points on the GT mesh, and get the attention weights in Fig. 9 (a). At each point, we show its distance to the mesh from the TSDF in Fig. 9 (b) and the mesh from the inferred occupancy in Fig. 9 (c), respectively. Fig. 9 (d) indicates where the former is smaller than the latter. The high correlation between Fig. 9 (a) and Fig. 9 (d) indicates that the attention network focuses more on the occupancy producing smaller errors to the GT surface. Instead, the red in Fig. 9 (e) indicates where the interpolated occupancy is larger than the inferred occupancy is not correlated to the one in Fig. 9 (a). The uncorrelation indicates that the attention network does not always focus on the larger occupancy input but the one with smaller errors, even without reconstructing surfaces.
(a) Attention on TSDF
(b) Error distances from the mesh on TSDF
Figure 9: Analysis on Attentions. (a) Map the attention weights on occupancies interpolated from TSDF. (b) The point-to-surface distances to the mesh reconstructed from TSDF. (c) The point-to-surface distances to the mesh reconstructed from the inferred occupancies. (d) Binary map indicating smaller errors (red) of TSDF. (e) Binary map indicating larger (red) interpolated occupancies than the inferred occupancies.
(c) Error distances to the mesh from inferred occupancies
(d) Binary map for smaller errors (red) of TSDF
(e) Binary map for larger (red) interpolated occupancies
We propose to learn neural implicit through volume rendering with attentive depth fusion priors. Our novel prior alleviates the incomplete depth at holes and the unawareness of occluded structures when using depth images as supervision in volume rendering. We also effectively enable neural networks to determine how much depth fusion prior can be directly used in the neural implicit. Our method can work well with depth fusion from either all depth images together or the ones available in a streaming way, using either known or estimated camera poses. To this end, our novel attention successfully learns how to combine the learned geometry and the depth fusion prior into the neural implicit for more accurate geometry representations. The ablation studies justify the effectiveness of our modules and training strategy. Our experiments on benchmarks with synthetic and real scans show that our method learns more accurate geometry and camera poses than the latest neural implicit methods.
This work was partially supported by a Richard Barber Interdisciplinary Research Award.
[97] Zihan Zhu, Songyou Peng, Viktor Larsson, Weiwei Xu, Hujun Bao, Zhaopeng Cui, Martin R. Oswald, and Marc Pollefeys. Nice-slam: Neural implicit scalable encoding for slam. In IEEE Conference on Computer Vision and Pattern Recognition, 2022.
[98] Zi-Xin Zou, Shi-Sheng Huang, Yan-Pei Cao, Tai-Jiang Mu, Ying Shan, and Hongbo Fu. Mononeuralfusion: Online monocular neural 3d reconstruction with geometric priors. CoRR, abs/2209.15153, 2022.