SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here,  $p_{OCC_v}^r$  is the occupancy probability of voxel  $v$  at resolution  $r$ ,  $y_{OCC_v}^r$  is the ground-truth occupancy label of voxel  $v$  at resolution  $r$ . By summing all losses, the total loss is:
$$
\mathcal {L} = \mathcal {L} _ {P} ^ {c} + \mathcal {L} _ {P} ^ {m} + \mathcal {L} _ {P} ^ {f} + \mathcal {L} _ {O C C} ^ {c} + \mathcal {L} _ {O C C} ^ {m} + \mathcal {L} _ {T S D F} ^ {f}. \tag {9}
$$
Datasets. Following previous works [8, 9, 11], we trained models on the ScanNet [40] dataset. ScanNet is a large-scale indoor scene dataset, which consists of 1613 RGB-D videos of 806 scenes. We follow the official train/eval/test split, where 1201 videos are used for training, 312 videos are used for evaluating and 100 videos are used for testing. Previous works evaluate the generalization performance of the models trained on ScanNet on TUM-RGBD [41] and ICL-NUIM [42] datasets. TUM-RGBD and ICL-NUIM datasets are also composed of RGB-D videos. Following previous works [11], 13 scenes of TUM-RGBD and 8 scenes of ICL-NUIM are used.
Evaluation metrics and protocol. Following conventional settings [9, 8, 11], we evaluate the performance using both 3D reconstruction metrics (e.g., Precision, Recall and F-Score) and 2D depth metrics (e.g., RMSE, Comp and Abs Rel). The detailed definition can be found in the supplementary material. Conventionally, Precision, Recall and F-Score are the three most important metrics. There
Table 2: Evaluation of the 2D depth map on ScanNet.
Methods	Abs Rel ↓	Abs Diff ↓	Sq Rel ↓	RMSE ↓	delta - 1.25 ↑
COLMAP [14]	0.137	0.264	0.138	0.502	0.834
MVDdepthNet [5]	0.098	0.191	0.061	0.293	0.896
GP-MVS [7]	0.130	0.239	0.339	0.472	0.906
DPSNet [18]	0.087	0.158	0.035	0.232	0.925
Atlas [8]	0.065	0.124	0.043	0.251	0.936
NeuralRecon [9]	0.065	0.106	0.031	0.195	0.948
VoRTX [11]	0.061	0.096	0.038	0.205	0.943
Ours	0.052	0.082	0.031	0.182	0.958
Table 3: Evaluation of the generalization. All models are trained on ScanNet datasets and tested on ICL-NUIM and TUM-RGBD.
Datasets	Methods	Acc ↓	Comp ↓	Prec ↑	Recall ↑	F-score ↑
ICL-NUIM	Atlas [8]	0.175	0.314	0.280	0.194	0.229
NeuralRecon [9]	0.215	1.031	0.214	0.036	0.058
VoRTX [11]	0.102	0.146	0.449	0.375	0.408
Ours	0.064	0.112	0.607	0.437	0.507
TUM-RGBD	Atlas [8]	0.208	2.344	0.360	0.089	0.132
NeuralRecon [9]	0.130	2.528	0.382	0.075	0.115
VoRTX [11]	0.175	0.314	0.280	0.194	0.229
Ours	0.072	0.140	0.564	0.248	0.343
are two different evaluation protocols. The original protocol directly compared the predicted 3D meshes with the ground-truth 3D meshes. Bozic et al.[10] notice that a more complete reconstruction can be penalized since some part of the ground-truth 3D meshes is incomplete. To avoid this problem, they applied an occupancy mask when evaluating models. In the following sections, we will conduct experiments to evaluate our framework under both protocols to demonstrate its effectiveness.
Training Details. We use the Adam optimizer [43] with  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.999$  and  $\epsilon = 10^{-8}$ . The learning rate is set to  $\alpha = 10^{-3}$  and is linearly warmed up from  $10^{-10}$  over 2000 steps. We trained our model for 500 epochs. We use the MnasNet-B1 [44] as the CNN backbone. We also apply the feature pyramid network [45] to extract features at different scales. The CNN backbone is fixed in the first 350 epochs and is finetuned with a learning rate  $\alpha = 10^{-4}$  in the last 150 epochs. The batch size is set to 4 and drops to 2 in the finetuning stage. Following previous methods [9, 11], we randomly drop out voxels to reduce the memory cost. In the training stage, we select 20 views for each video and the selection strategy is following [9]. To have a fair comparison, the voxel size of the fine/medium/coarse level is set to  $4\mathrm{cm} / 8\mathrm{cm} / 16\mathrm{cm}$  and the TSDF truncation distance is set to triple the voxel size. Training our model takes about 90 hours on a single Nvidia RTX 3090 graphic card.
To evaluate the performance of our model, we first compare the performance of our model and existing state-of-the-art methods on the large-scale indoor scene dataset ScanNet. The results are reported in Table. 1. As shown in the table, our model achieves state-of-the-art performance on almost all metrics. For example, our model improves the F-score from 0.641 to 0.663 and the Precision from 0.708 to 0.748 when evaluated based on the first protocol [11]. Based on the Second evaluation protocol [10], our model can also increase the F-score from 0.703 to 0.712 and the Precision from 0.767 to 0.791. The design of our model aims to find better classification boundaries, which can lead to precise surface voxel prediction. The significant improvement in Precision demonstrates the effectiveness of our design and shows that our model can make much more precise predictions than existing methods.
Following previous works, we also evaluate our model with 2D depth map metrics. Since volumetric-based methods directly predict TSDF volume instead of depth maps, we render the predicted 3D meshes to the image planes to get the depth maps. The results are reported in Table. 2. The results
Table 4: Evaluation of each component of our model. Here, "Branch" indicates using separate branches to predict TSDF and occupancy. "IOA" indicates using the inner-outer aware coarse-to-fine strategy and loss.
Branch	IOA	Acc ↓	Comp ↓	Prec ↑	Recall ↑	F-score ↑
X	X	0.054	0.090	0.708	0.588	0.641
✓	X	0.054	0.086	0.712	0.607	0.654
X	✓	0.044	0.092	0.743	0.583	0.652
✓	✓	0.043	0.090	0.748	0.597	0.663
Figure 3: Visualizations of outer-surface, surface, and inner-surface voxels using t-SNE [46]. Here, the outer-surface, surface, and inner-surface voxels are colored by red, green and blue, respectively. Four subfigures are the results of four different scenes.
show that our model can decrease the relative error by  $14.7\%$  from 0.061 to 0.052 and get a more accurate depth map.
To evaluate the generalizability of our model, we also conduct experiments on two other datasets ICL-NUIM and TUM-RGBD. Models are trained on ScanNet and tested on ICL-NUIM and TUM-RGBD. The results are reported in Table. 3. On the ICL-NUIM dataset, the Precision is improved from 0.449 to 0.607 and the F-score is improved from 0.408 to 0.507. On the TUM-RGBD dataset, the Precision is improved from 0.280 to 0.564 and the F-score is improved from 0.229 to 0.343. Compared to existing methods, our model generalizes well on both datasets. A possible reason is that existing methods overfit the training set to find the complicated classification boundary between surface and non-surface voxels. Instead, our model finds the classification boundaries between inner-surface and outer-surface voxels, between surface and inner-surface voxels, and between surface and outer-surface voxels, which is much easier and thus alleviates overfitting.
In addition to comparing our model with existing methods, we also conduct an ablation study to further explore the contribution of each part of our model. To this end, we design two variant models. The first one is using separate occupancy branches without the inner-outer aware coarse-to-fine strategy and loss. The second one uses the inner-outer aware coarse-to-fine strategy and loss but extracts occupancy and TSDF using a shared 3D CNN backbone. The results are reported in Table. 4. We separate the occupancy and TSDF branches since they may interfere with each other with a shared 3D CNN backbone. Compared to the basic model, using separate occupancy and TSDF branches can slightly improve all metrics. The TSDF prediction and occupancy prediction are both improved which demonstrates the effectiveness of this design. The inner-outer aware coarse-to-fine strategy and loss aim to better classify the surface voxels. Compared to the basic model, using inner-outer aware coarse-to-fine strategy and loss leads to significant improvement in Precision. This result shows that using the new strategy and loss can help the model make more precise predictions and better classify the surface voxels. Finally, the full IOAR model can further improve the performance, which shows that the two components can collaborate well.
4.3 Further Analysis
In this section, we further analyze our model with qualitative and quantitative results. First, we answer an important question: are inner-surface voxels really quite different from outer-surface voxels?
Figure 4: Visualization of 3D meshes reconstructed by our model and existing methods. From left to right are the meshes reconstructed by Atlas, NeuralRecon, VoRTX and our model. And the most right column contains the ground-truth meshes. We enlarge the detailed area in the red box to display it more clearly.
Second, we show some visualization results to compare the quality of 3D meshes reconstructed by our model and other methods.
Are inner-surface voxels really quite different from outer-surface voxels? This is an important question since grouping inner-surface and outer-surface voxels into the same class will force the model to waste its capability only when there is a gap between the inner-surface and outer-surface voxels (i.e., the inner-surface voxels are quite different from the outer-surface voxels). To answer this question, we conduct both qualitative and quantitative experiments. First, we visualize the features of surface, inner-surface, and outer-surface voxels in Figure 3. As we can see, the features of inner-surface and outer-surface voxels are quite different and the features of surface voxels usually distribute between inner-surface and outer-surface voxel features. Therefore, it is difficult for a classifier to distinguish non-surface voxels from surface voxels, which leads to a complicated classification boundary. This supports the motivation of our model. Second, we measure the similarity between outer-surface voxels and inner-surface voxels (denoted as  $s_{oi}$ ), the similarity between outer-surface voxels and surface voxels (denoted as  $s_{os}$ ) and the similarity between inner-surface voxels and surface voxels (denoted as  $s_{is}$ ). Since there are a large number of voxels in each volume, measuring
the similarity between all pairs of voxels is computationally impossible. Instead, we compute the feature prototypes of surface, non-surface, inner-surface and outer-surface voxels by averaging the features. Then, we measure the similarity between each class based on the cosine similarity between prototypes. We average the similarity over all test scenes and get  $s_{oi} = 0.6956$ ,  $s_{os} = 0.8933$  and  $s_{is} = 0.8829$ . This also demonstrates that inner-surface voxels are quite different from outer-surface voxels. We also analyzed the accuracy of discriminating inner-surface, outer-surface and surface voxels. The results show that only a few inner-surface voxels are misclassified as outer-surface voxels and only a few outer-surface voxels are misclassified as inner-surface voxels. This also supports our assumption that outer-surface voxels are quite different from inner-surface voxels since the model can easily distinguish them in most cases. For more details about this experiment, please refer to our supplementary material.
Visualization. To have an intuitive comparison with existing works, we visualize the 3D meshes reconstructed by our model and other methods in Figure. 4. As we can see, the reconstructed meshes of our model are more precise and complete than existing methods. Taking the first scene as an example, our model can precisely reconstruct a flat desktop while the desktop reconstructed by Atlas and NeuralRecon are uneven. And compared with VoRTX, we can reconstruct the table leg which is lost in their result.
In this work, we propose the inner-outer aware reconstruction (IOAR) model for monocular 3D scene reconstruction. Different from existing methods, IOAR can classify the inner-surface voxels and outer-surface voxels, which contributes to precise occupancy prediction and TSDF prediction. To achieve this goal, we design new inner-outer aware coarse-to-fine strategy and loss, which leads the model to learn to classify the inner-surface voxels and outer-surface voxels. In addition, to avoid the mutual interference between TSDF prediction and occupancy prediction, IOAR separates the occupancy branches from the TSDF branches. Thanks to these effective designs, IOAR achieves state-of-the-art performance on the large-scale indoor scene dataset ScanNet and generalizes well on unseen scenes.
This work was supported partially by the NSFC(U21A20471,U1911401), Guangdong NSF Project (No. 2023B1515040025, 2020B1515120085).
[1] C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós, "ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM," IEEE Transcation Robotics, vol. 37, no. 6, pp. 1874-1890, 2021.
[2] L. von Stumberg, V. Usenko, and D. Cremers, "Direct sparse visual-inertial odometry using dynamic marginalization," in International Conference on Robotics and Automation, pp. 2510-2517, IEEE, 2018.
[3] J. L. Schonberger and J. Frahm, "Structure-from-motion revisited," in Conference on Computer Vision and Pattern Recognition, pp. 4104-4113, IEEE, 2016.
[4] A. Düzçeker, S. Galliani, C. Vogel, P. Speciale, M. Dusmanu, and M. Pollefeys, “Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion,” in Conference on Computer Vision and Pattern Recognition, pp. 15324–15333, IEEE, 2021.
[5] K. Wang and S. Shen, "Mvdepthnet: Real-time multiview depth estimation neural network," in International Conference on 3D Vision, pp. 248-257, IEEE, 2018.
[6] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, "Mvsnet: Depth inference for unstructured multiview stereo," in European Conference on Computer Vision, vol. 11212, pp. 785-801, Springer, 2018.
[7] Y. Hou, J. Kannala, and A. Solin, "Multi-view stereo by temporal nonparametric fusion," in International Conference on Computer Vision, pp. 2651-2660, IEEE, 2019.
[46] L. van der Maaten and G. Hinton, "Visualizing data using t-sne," Journal of Machine Learning Research, vol. 9, no. 86, pp. 2579-2605, 2008.