Inner-Outer Aware Reconstruction Model for Monocular 3D Scene Reconstruction
Authors: Yu-Kun Qiu
Guohao Xu
Wei-Shi Zheng
Venue: NeurIPS 2023 poster (2023)
Keywords: 3D reconstruction

Abstract: Monocular 3D scene reconstruction aims to reconstruct the 3D structure of scenes based on posed images. Recent volumetric-based methods directly predict the truncated signed distance function (TSDF) volume and have achieved promising results. The memory cost of volumetric-based methods will grow cubically as the volume size increases, so a coarse-to-fine strategy is necessary for saving memory. Specifically, the coarse-to-fine strategy distinguishes surface voxels from non-surface voxels, and only potential surface voxels are considered in the succeeding procedure. However, the non-surface voxels have various features, and in particular, the voxels on the inner side of the surface are quite different from those on the outer side since there exists an intrinsic gap between them. Therefore, grouping inner-surface and outer-surface voxels into the same class will force the classifier to spend its capacity to bridge the gap. By contrast, it is relatively easy for the classifier to distinguish inner-surface and outer-surface voxels due to the intrinsic gap. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. IOAR explores a new coarse-to-fine strategy to classify outer-surface, inner-surface and surface voxels. In addition, IOAR separates occupancy branches from TSDF branches to avoid mutual interference between them. Since our model can better classify the surface, outer-surface and inner-surface voxels, it can predict more precise meshes than existing methods. Experiment results on ScanNet, ICL-NUIM and TUM-RGBD datasets demonstrate the effectiveness and generalization of our model. The code is available at https://github.com/YorkQiu/InnerOuterAwareReconstruction.

Inner-Outer Aware Reconstruction Model for Monocular 3D Scene Reconstruction
Yu-Kun Qiu'
Guo-Hao Xu'
Wei-Shi Zheng 2.2
School of Computer Science and Engincering, Sun Yat-sen University, China Key Laboratory of Machine Intelligence and Advanced Computing, Ministry of Education, China {qiuyk, xmah23)@mail2 wszhengGieee.oog
0maal2maaa1l... sysu. edu.
Abstract
Monocular 3D scene reconstruction aims reconstruct the 3D structure of scenes based on posed images. Recent volumetric-base methods directly predict the truncated signed distance function (TSDF) volume and have achieved promising results. The memory cost of volumetric-basee methods will grow cubically as the volume size increases, memory. Specifically, the coarse-to to fine strategy distinguishes surface voxels from non-surface voxels, and only potential surface voxels considered in the succeeding procedure. However, the non-surface voxels have various features, and in particular, the voxels the inner side of the surface are quite different from those on the outer side since there exists an intrinsic gap between them. Therefore, grouping inner-surface and outer-surface voxels into the same class will force the classifier to spend its capacity to bridge the gap. By contrast, it is relatively easy for the classifier distinguish inner-surface and outer-surface voxels due to the intrinsic gap. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. IOAR explores new coarse-to-fine strategy to classify outer-surface, inner-surface and surface voxels. In addition, IOAR separates occupancy branches from TSDF branches to avoid mutual interference between them. Since our model can better classify the surface, outer-surface and inner-surface voxels, can predict more precise meshes than existing methods. Experiment results on ScanNet, ICL-NUIM and TUM-RGBD datasets demonstrate the effectiveness and generalization of our model. The code is available lhttss
coarse-to-Bi strategy is necessary for saving
Introduction
Monocular 3D Reconstruction is fundamental task in computer vision and is the basis for var- ious applications like robotics, augmented/virtual reality and autonomous navigation. It aims at reconstructing the 3D structure of scenes or objects based on sequence of monocular RGB images. Although camera poses can be estimated accurately with state-of-the-art SLAM 21 or SiM systems
33, this task is still very challenging since the complexity of the environment. For example, indoor scenes can contain various objects with different textures under diverse illumination.
Researchers have studied this challenging problem for many years, and conventional methods solve this problem in multi-view depth manner [4] 5 7 They predict the depth map for each image and then fuse these depth maps to generate 3D meshes. Recently, some researchers 8 9 I0 12
attempted to regress the 3D surface of the entire scene directly and have achieved promising results. They first extract image features from different views and then back-project them to form volume
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

View
View
damilication boundary
View
View
Voaat
editing nethods
surface vusii
Youd
Inne--sorface
veuch
(8)
(b)
(c)
Figure 1: (a) Illustration of simple scene with voxels (denoted by capital letters, e.g., A, B and C). (b) The observations of voxel A, B and C in four different views. (c) The classification boundaries find by existing methods and our model.
features. Finally, they utilize a 3D backbone to predict TSDF volume which can be converted 3D mesh using the marching cubes algorithm [13]. As the volume size increases, the consumed memory will grow cubically. Therefore, existing methods apply coarse-to-fine strategy to reduce memory costs. Specifically, they train a classifier for each level to classify whether voxel is a surface voxel or not, and only the potential surface voxels will be selected and passed the next level. However, the voxels on the inner side of the surface (inner-surface voxels) are quite different from the voxels on the outer side of the surface (outer-surface voxels). Due to the occlusion of the surface, the observations of inner-surface voxels are usually similar to the surface voxels. On the other hand, the outer-surface voxels are not occupied, and thus their observations from different views are surfaces of different instances. Hence, there is an intrinsic gap between the inner-surface voxels and outer-surface voxels. We give an example in Figure. Figure. (a) is a simple scene with only a circular object in the center of room. And Figure. (b) shows the observation of voxels A. B and C in four different views. As we can sec, observations of inner-surface voxel A and outer-surface voxel C in views and 4 are quite different (i.e., object surface wall surface). In fact, the inner-surface voxel A is more similar to the surface voxel since their observations under all views are different parts of the object's s surface. Due the large gap between inner-surface voxels and outer-surface voxels, it is improper to group all of them into one class, which requires the model to spend its capacity to bridge the gap. What is worse, this lead to overfitting. As shown in Figure. (c), previous methods have to learn complicated classification boundaries to classify the surface voxels and non-surface voxels.
By contrast, there exists an intrinsic gap, classifying the inner-surface voxels and outer-surface voxels into two classes is relatively easy, meanwhile avoiding the waste of model capacity and alleviating the risk of overfitting. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. Different from existing methods, IOAR explores a new coarse-to-fing strategy in which classifier can classify whether voxel is surface, inner-surface outer-surface voxel. In addition, we design separate TSDF and occupancy branches to avoid the mutual interference between TSDF prediction and occupancy prediction. Finally, we apply an inner-outer aware loss to supervise the inner-outer aware occupancy branch to learn to classify the inner-surface, outer-surface and surface voxels. Since it is easier to find classification boundaries between surface voxels and inner-surface voxels and between surface voxels and outer-surface voxels (Figure (c)), IOAR can make more accurate predictions than previous methods. Experiments show that IOAR improves the F-score from 64.1 to 66.3 and the Precision from 70,8 to 74.8 on ScanNet. Further experiments on ICL-NUIM and TUM-RGBD datasets show that IOAR has good generalization performance.
Related Works
Depth-based 3D reconstruction methods reconstruct the 3D structure by two steps. First, estimate the depth of each selected video frame. Second, fuse these depths to reconstruct the 3D structure. Conventional methods [14 [S] 16 I7] typically use patch-matching based photometric consistency to estimate the depth. With the development of deep learning, many researchers paid attention to estimating accurate and dense depth maps using deep neural networks (. 18
20 21].
MVSNet 66 extracts image features from multiple views and then builds a 3D cost volume based on the variance between reference view and source views. Then MVSNet predicts the depth map of the reference view based on the cost volume. MVDepthNet (5) builds the cost volume based on

---

F
V
Branch
Fusion
back- project
Tnner-Outer Aware
coarse-to-fine sparsity
Occupancy Branch
Input: posed Images
V
ISDF Branch
Fusion
CNN Backbone
Inner-Outer Aware Occupancx Branch
project
coarse-to-fine sparsify
F
v
F Fusion
back-
Branch
project
Figure 2: The overall pipeline of model. Given series of posed images, first extract the coarse/medium/fine-leee image features F Fm. Ff using a CNN backbone. Then, image features are pack-projected to form feature volumes V,s Va Vi( fusion module is utilized to fuse feature volumes from different views. The fused features Ve, VTM v are then input into the TSDF or occupancy branch to predict TSDF or occupancy volume. The TSDF or occupancy features and logits are utilized to predict TSDF or occupancy volume in the next level.
pixel intensity and applies an encoder-decoder architecture to predict depth in multiple resolutions simultaneously. GP-MVS (7) utilizes the Gaussian process to leverage information from previous latent-space encodings to predict precise depth maps. Instead of directly predicting depth maps, Neural RGB-D [22] estimates Depth Probability Volume (DPV) for each input frame. They design a deep network based on Bayesian filtering theory, which accumulates the DPVs across frames predict accurate depth maps. DeepV2D 2X proposes differentiable flow-to-depth layer and estimates the depth maps based on the optical flow. DeepVideoM's 4 is an LSTM based model, which explores the scene geometry information from previous steps. SimpleRecon [9] injects readily available metadata into the cost volume and achieves good performance without the help of 3D convolutions. After estimating the depth, methods like Poisson surface reconstruction [24] 25],
Delaunay Triangulation [26] and TSDF Fusion [27 28
29
30
can be used to generate 3D mesh
based on inconsistent single-view depths.
Volumetric-based 3D reconstruction methods directly regress a volumetric data structure to solve the problem in end-to-end manner. SurfaceNet (31 builds colored voxel cubes (CVC) based on images and explores 3D convolutional networks to predict a surface occupancy volume. Atlas back-projects the selected 2D image features into 3D feature volume. Taking this volume as input, a 3D U-Net predicts a TSDF volume in coarse-to- to-eine manner. Each voxel in the TSDF volume describes the truncated signed distance the nearest surface and the TSDF volume can be converted to 3D mesh using the marching cubes algorithm [33] NeuralRecon [9] achieves real-time 3D reconstruction by reconstructing the scene in each local window and then merges the result using a GRU 32) module. Instead of averaging the contribution of each image feature, TransformerFusee 10 pays attention to the most relevant images using transformer mechanism. VoRTX [IT] also applies transformer mechanism to aggregate image features. In addition, VoRTX weights the contribution of image features for each voxel based on the projective occupancy. 3D-Former [12] designs an 3D Transformer to replace the 3D CNN in previous works and clearly improves the performance. Theoretically. 3D-Former can be utilized as the 3D backbone of any volumetric-base 3D reconstruction method, and our method can significantly improve performance with its help. However, have a fair comparison with existing works, we still use the 3D CNN as the 3D backbone.
Neural implicit representation of 3D structure. Some researchers 33 34
36 aim to find
neural implicit representation of the 3D structure. Mescheder et al. (33 present OccupancyNetworr which learns a mapping from 3D coordinates and observations to the probability of occupancy. Park al. 34 propose DeepSDF which predicts the SDF value of each point. NSVF (35] progressively sparsifies the space and finds mapping in the sparse space, thus highly reducing the computational cost. NeRFusion (36] combines the advantages NeRF and -ased fusion methods to achieve efficient large-scale reconstruction.

---

In this work, we propose the inner-outer aware reconstruction (IOAR) model which aims to predict meshes more precisely by finding better classification boundaries between surface voxels and inner- surface voxels and between surface voxels and outer-surface voxels. IOAR explores new coarse-to- fine strategy in which the classifier can classify the inner-surface, outer-surface, and surface voxels. In addition, IOAR avoids the mutual interference between TSDF and occupancy prediction by using separate branches to predict TSDF and occupancy volume. Under the supervision of inner-outer aware loss, the occupancy branch can predict the potential surface more accurately. Extensive experiments on several datasets demonstrate the effectiveness and generalization of our IOAR.
Inner-Outer Aware Reconstruction
Given set of N posed images (In) of a scene with corresponding camera intrinsics [K LN and camera extrinsics {Pa)am our model aims to predict a TSDF volume VISDF E
[-1,1]4%xXR x x This TSDF volume will further be converted 3D mesh using the marching
cubes algorithm [33. We illustrate the pipeline of our model in Figure. 2 The posed images are input into CNN backbone to extract image features at different scales. The feature of the image in view at coarse/medium//fngg level are indicated as F F.. Then, these features are back-projected to form feature volumes V Vim V and a fusion module fuses volumes from N views to generate fused feature Vo, V V Finally, the fused features are input into the TSDF or occupancy branch to predict the TSDF or occupancy volume. In the medium and fine level, the TSDF/occupancy features and logits from the previous level are utilized to predict the TSDF/occupancy volume at current level.
3.1 Feature Volume Construction
The posed images are input into a 2D CNN backbone F(-) to extract 2D features at different scales. Using F(-), we can get the image feature of I, at coarse/medium/fine level n) F. F = F(T)) focus
Here, FC RCC, x x Fm RCT, " xw" F! RC x'W' For simplicity, on a single resolution r (r E (c, m, Then these image features are back-projected to form feature volumes. Specifically, using corresponding camera intrinsic K,, and camera extrinsics P, we can form the feature volume V R C. XN,
XNTXNNNN XN; [
ITK PR
K
Here, II is the perspective mapping, : the slice operator and NT x Ny x N is the volume size in resolution r. Eq. (I) maps the voxel coordinates (i, k) in the world space to the pixel coordinates (i,5) in the image space and assigns the image features to the corresponding voxel features. This means that all voxels along a ray are filled with the same features of the corresponding pixel.
Then we need to fuse the feature volumes among all views. A straightforward method is averaging all feature volumes (i.e., V" = N E ViD. However, different views provide different information when predicting TSDF or occupancy of voxels. Therefore, assigning larger weights to views that provide information that contributes to TSDF or occupancy prediction results in better performance. Following VoRTX [II, we use transformer to fuse the feature volumes and assign weights based on projective occupancy. First, a transformer (3 (T(-) is used to augment each single-view feature volume with information from other relevant views:
0 05. B = (of ......
Here, of, is the feature of a voxel in volume Vi (i.e., oh E Vg). Then, we can predict the projective occupancy Or E R-N,
O o(W,), W. = (' ( .,,))))))
Here, (-(-) is an MLP and (-)) is a sigmoid function. The corresponding ground-truth (O"), is equal if the voxel o view n with resolution is close enough to the surface. The detailed definition of OF can be found in the supplementary material. We concatenate zero to W to indicate voxel is not close to surface under all N views and apply softmax to compute weight vector WT e N+1

---

Then the weighted voxel feature of voxel is computed by o" = Wo ,,,,,,, 0]. After weighting all voxels, we can get the fused feature volume V E IR X.rxNNxx,,, NExANxNE
3.2 TSDF and Inner-Outer Aware Occupancy Branch
Taking the fused feature volume V as input, the TSDF and inner-outer aware occupancy branch predict the TSDF value and occupancy probability for each voxel. The TSDF/occupancy branch at each level consists of 3D CNN backbone and TSDF/oceupancy head.
The procedure at the coarse level. We denote the 3D CNN backbone of the TSDF/occupancy branch (F () The fused feature volume Ve is input into each 3D CNN backbone
and generates the TSDF feature FîSDE E R NN Ny N and occupancy feature Föcc e R.C.u x x x by:
SISDF (VsD FF = (occ(V
The TSDF head H (-) is a 3D convolutional layer with the output channel equal 1. Different from previous works, since we aim to classify the surface, inner-surface and outer-surface voxels, the occupancy head (-)c(() is 3D convolutional layer with an output channel equal to 3. We can get the TSDF logits sSDD =E R 1IN and occupancy logits locc E xxXxN at the coarse level by:
lSDD = HySDr (FisDr), Foce HoCce
(5)
Inner-outer aware coarse-to-fine strategy. Before introducing the procedure in the medium and fine level, we first introduce the inner-outer aware coarse-to-fine strategy utilized by the inner-outer aware occupancy branch. Given the ground-truth TSDF VISDP R.N: XN,xNN
can define surface voxels, inner-surface voxels and outer-surface voxels as follows. A voxel v is surface voxel if is very close to the surface (i.e., this voxel is in the truncated range of a surface (-1 (WTsDF) 1)). A voxel v is an inner-surface voxel if it is on the inner side of the surface (i.e., this voxel is not in the truncated range of any surface and its truncated distance is 1 ((ysspr) is an outer-surface voxel if it is on the outer side of the surface (i.e., this voxel is not in the truncated range of any surface and its truncated distance is -1 (WTSDF); -1)) Then based on the definition of surface voxels, inner-surface voxels and outer-surface voxels, the occupancy ground-truth Voce E TRNN,NN x.,,NNNN X can be defined as:
1)). A voxel
0 if is a inner-surface voxel (i.e., (VTSDF), = 1),
1 if u a surface voxel (i.e, -1 < (VTSDF)u < <1) (yocc) =
(6)
2 if is an outer-surface voxel (i.e., (FTSDF) = -1).
Given the occupancy probability at the coarse level pôcc = softmax (18gg) we can get the occupancy state at coarse level Soco = arg max(Foce) E {0, 1, NF NN, Intuitively, we can get an occupancy mask maske at coarse level where mask; = I[((Scc), = 1). Here, I(c) equals to 1 if condition e is satisfied. With the help of the occupancy mask, we can sparsify the dense volume and reduce memory and computation costs. All sparse operations in our work are based on torchsparse [38] Different from cxisting methods, our coarse-to-fine -ine strategy makes it casier for the classifier to find better classification boundaries which leads to precise mesh prediction. For simplicity, we still use dense volumes in the following description.
The procedure at the medium/fine level. Different from the coarse level, the features from the previous level are utilized for better prediction in the medium and fine levels. To utilize the features in the previous level, we expand TSDF/occupancy features and logits in the previous level to have the same size as the current level. The expanded TSDF features F E C - x N x N x xN
and logits SSSDE
I x XN." N. are concatenated with fused feature volume V at the
current level (medium level) to form medium level aggregated TSDF feature volume TSDE E
R RCC... + +C N xN, x which is used for extracting medium level TSDF features FTSDF
R C N xN N
Fm TSDF TTSDr (TSDD((V'DD)) (VTDD)) VISD cat(fFispr. V (ssDP).
Vm
Im Here, Nm 2 No 2Ng, Nm 2Ng. The procedure of generating TSDF logits SSDE is the same as coarse level using TSDF head HT Similarly, we can extract the medium-level
5

---

Table 1: Evaluation of the 3D meshes on ScanNet. The upper part follows the evaluation protocol in
VoRTX 11 and the lower part follows the evaluation protocol in TransformerFusion [00
Methods ViepVidM 4
Acc +
Comp
Prec
Recall f
F-score
0.079
0.133
0.521
0,454
0,474
NeuralRecon
0.054
0.128
0.684
0.479
0.562
Atlas [8]
0.068
0.098
0.640
0.539
0.583
VoRTX
0.054
0.090
0.708
0.588
0.641
Ours
0.043
0.090
0.748
0.597
0.663
COLMAP
0.102
0.119
0.509
0.474
0.489
MVDepthNet 5
0.129
0.083
0.443
0,487
0.460
GP-MVS 7
0.129
0.080
0.453
0,510
0.477
DPSNet [T8
0.119
0.076
0.474
0.519
0.492
ESTDepth
39
0.127
0.075
0.456
0.542
0.491
Ved
0.107
0.069
0.541
0.592
0.563
NeuralRecon
0.051
0.091
0.630
0,612
0.619
TransformerFusion
10
0.055
0.083
0.728
0.600
0.655
SimpleRecon
19
0.055
0.061
0.686
0.658
0,671
Atlas
0.072
0.076
0.675
0.605
0.636
VoRTX
|
0.043
0.072
0.767
0.651
0.703
Ours
0.037
0.072
0.791
0.650
0.712
occupancy feature Föce E IR Cout x N X.N X and the rest of the procedure is the same as the coarse level. A detailed description of the expansion operation is introduced in the supplementary material.
At the fine level, all voxels are treated as potential surface voxels. Therefore, we do not set fine-level occupancy branch. Except for this, the procedure of the fine level the same as that of the medium level.
3.3 Training Loss
The loss function be divided into three parts, the projective occupancy loss, the inner-outer occupancy loss, and the TSDF loss. Following previous methods, the projective occupancy loss Cp is binary cross-entropy loss [1] and the TSDF loss CISDE a distance loss [8] The inner-outer
aware occupancy loss can be computed by:
Coco NI x
Σ log(Poco, Mo0..
(8)
N NT
"Eroce
Here, Pocc, is the occupancy probability of voxel resolution r, Focc, is the ground-truth occupancy label voxel v at resolution r. By summing all losses, the total loss is:
L Ci + CT + C/p + foce + Cocc + C/SDE
(9)
Experiments
Datasets. Following previous works 8
we trained models the ScanNet 44) dataset.
ScanNet is a large-scale indoor scene dataset, which consists of 1613 RGB-D vidcos of 806 scenes. We follow the official train/eval/test split, where 1201 videos are used for training, 312 vidcos used for evaluating and 100 videos are used for testing. Previous works evaluate the generalization performance of the models trained on ScanNet TUM-RGBD and ICL-NUIM datasets are also composed of RGB-D videos. Following previous works [1], 13 scenes of TUM-RGBD and 8 scenes of ICL-NUIM used.
TUM-RGBD 41 and ICL-NUIM 42) datasets.
Evaluation metrics and protocol. Following conventional settings 9
we evaluate performance using both 3D reconstruction metrics (e.g., Precision, Recall and F-Score) and 2D depth metrics (e.g., RMSE, Comp and Abs Rel). The detailed definition can be found in the supplementary material. Conventionally, Precision, Recall and F-Score are the three most important metrics. There

---

Table 2: Evaluation of the 2D depth map on ScanNet.
Methods
Abs Rel +
Abs Diff
Sq Rel
RMSE
della - 1.25
COLMAP [14]
0.137
0.264
0.138
0.502
0.834
MVDepthNet
0.098
0.191
0.061
0.293
0.896
GP-MVS 7
0.130
0.239
0.339
0.472
0,906
DPSNet [8]
0.087
0.158
0.035
0.232
0.925
Atlas
0.065
0.124
0.043
0,251
0,936
NeuralRecon 9
0.065
0.106
0.031
0.195
0,948
VoRTX [|]
0.061
0.096
0.038
0.205
0,943
Ours
0.052
0.082
0.031
0.182
0.958
Table Evaluation of the generalization. All models are trained on ScanNet datasets and tested ICL-NUIM and TUM-RGBD.
Datasets
Methods
Acc +
Comp
Prec
Recall f
F-score 1
Atlas 8
0.175
0.314
0.280
0.194
0.229
NeuralRecon 9
0.215
1.031
0.214
0.036
0.058
ICL-NUIM
VoRTX 11
0.102
0.146
0.449
0.375
0.408
Ours
0.064
0.112
0.607
0.437
0.507
Atlas 8
0.208
2.344
0.360
0.089
0.132
NeuralRecon 9
0.130
2.528
0.382
0.075
0.115
TUM-RGBD
VoRTX 11
0.175
0.314
0.280
0.194
0.229
Ours
0.072
0.140
0.564
0.248
0.343
two different evaluation protocols. The original protocol directly compared the predicted 3D meshes with the ground-truth 3D meshes. Bozic et al. [10] notice that more complete reconstruction can be penalized since some part of the ground-trun truth 3D meshes is incomplete. To avoid this problem, they applied an occupancy mask when evaluating models. In the following sections, we will conduct experiments evaluate our framework under both protocols demonstrate its effectiveness.
Training Details. We use the Adam optimizer [43 with 31 = 0.9, 3g = 0.999 and e = 10-8 The learning rate is set
10-3 and is linearly warmed up from 10-10 over 2000 steps. We trained our model for 500 epochs. We use the MnasNet-B1 [4 as the CNN backbone. We also apply the feature pyramid network [45] to extract features different scales. The CNN backbone is fixed in the first 350 epochs and is finetuned with learning rate = 10-4 in the last 150 epochs. The and drops to 2 the finetuning stage. Following previous methods I.
batch size is set randomly drop out voxels to reduce the memory cost. In the training stage, we select 20 views for each video and the selection strategy following (9. To have fair comparison, the voxel size the fine/medium/coarse level is set to 4cm/ 6cm and the TSDF truncation distance is set to triple the voxel size. Training our model takes about hours on single Nvidia RTX 3090 graphic card.
4.1 Evaluation
To evaluate the performance of our model, we first compare the performance of our model and existing state-of-the-ar methods on the large-scale indoor scene dataset ScanNet. The results are reported in Table. [ As shown in the table, our model achieves state-of-the -hee-of-t performance all metrics. For example, our model improves the F-score from 0.641 I to 0.663 and the Precision from 0.708 to 0. 748 when evaluated based the first protocol [II. Based on the Second evaluation protocol 10), our model can also increase the F-score from 0.703 to 0.712 and the Precision from 0.767 to 0.791. The design of our model aims to find better classification boundaries, which can lead to precise surface voxel prediction. The significant improvement in Precision demonstrates the effectiveness of our design and shows that model make much more precise predictions than existing methods.
almost
Following previous works, we also evaluate our model with 2D depth map metrics. Since volumetric- based methods directly predict TSDF volume instead of depth maps, we render the predicted 3D meshes to the image planes to get the depth maps. The results reported in Table. 2 The results

---

Table 4: Evaluation of each component of our model. Here, "Branch" indicates using separate
branches to predict TSDF and oceupancy. "IOA" indicates using the inner-outer aware coarse-to- strategy and loss.
Branch
IOA
Acc 4
Comp 4
Prec 1
Recall f
F-score f
X
0.054
0.090
0.708
0.588
0.641
x
0.054
0.086
0.712
0.607
0.654
X
0.044
0.092
0.743
0.583
0.652
0.043
0.090
0.748
0.597
0.663
Figure 3: Visualizations of outer-surface, surface, and inner-surface voxels using t-SNE 46 Here, the outer-surface, surface, and inner-surface voxels are colored by red, green and blue, respectively. Four subfigures are the results of four different scenes.
show that our model can decrease the relative error by 14.7% from 0.061 to 0.052 and get a more accurate depth map.
To evaluate the generalizability of our model, we also conduct experiments on two other datasets ICI.- NUIM and TUM-RGBD. Models are trained on ScanNet and tested on ICL-NUIM and TUM-RGBD. The results are reported in Table. g the ICL-NUTM dataset, the Precision improved from 0.449 to 0.607 and the F-score is improved from 0.408 to 0.507. On the TUM-RGBD dataset, the Precision improved from 0.280 to 0.564 and the F-score is improved from 0.229 to 0.343. Compared to existing methods, our model generalizes well on both datasets. A possible reason is that existing methods overfit the training set to find the complicated classification boundary between surface and and
non-surface voxels. Instead, our model finds the classification boundaries between outer-surface voxels, between surface and inner- surface voxels, and between surface and outer-surface voxels, which is much easier and thus alleviates overfitting.
4.2 Ablation Study
In addition to comparing our model with existing methods, we also conduct an ablation study to further explore the contribution of each part of our model. this end, we design two variant models. The first one is using separate occupancy branches without the inner-outer aware coarse-to-fine strategy and loss. The second one uses the inner-outer aware coarse-to- fonfine strategy and loss extracts occupancy and TSDF using shared 3D CNN backbone. The results are reported in Table. [ We separate the occupancy and TSDF branches since they may interfere with each other with a shared 3D CNN backbone. Compared to the basic model, using separate occupancy and TSDF branches can slightly improve all metrics. The TSDF prediction and occupancy prediction are both improved which demonstrates the effectiveness of this design. The inner-outer aware coarse-to- co-re-to-ffine strategy and loss aim to better classify the surface voxels. Compared to the basic model, using inner-outer aware coarse-to-fine strategy and loss leads significant improvement in Precision. This result shows that using the new strategy and loss can help the model make more precise predictions and better classify the surface voxels. Finally, the full IOAR model can further improve the performance, which shows that the two components can collaborate well.
4.3 Further Analysis
In this section, we further analyze our model with qualitative and quantitative results. First, we answer an important question: are inner-surface voxels really quite different from outer-surface voxels?

---

Atlas
NeuralReoon
VORIX
Ours
Ground
Figure 4: Visualization of 3D meshes reconstructed by our model and existing methods. From left to right are the meshes reconstructed by Atlas, NeuralRecon, VoRTX and our model. And the most right column contains the ground-truth meshes. enlarge the detailed area in the red box to display it more clearly.
Second, we show some visualization results to compare the quality of 3D meshes reconstructed by our model and other methods.
Are inner-surface voxels really quite different from outer-surface voxels? This is an important question since grouping inner-surface and outer-surface voxels into the same class will force the model to waste its capability only when there a gap between the inner-surface and outer-surface voxels (i.e., the inner-surface voxels are quite different from the outer-surface voxels). To answer this question, we conduct both qualitative and quantitative experiments. First, we visualize the features of surface, inner-surface, and suter-surface voxels Figure. 3 As can see, the features of inner- surface and outer- sureres voxels are quite different and the features of surface voxels usually distribute between surf-eeuffae and outer-surface voxel features. Therefore, it is difficult for classifier to distinguish non-surface voxels from surface voxels, which leads to complicated classification boundary. This supports the motivation of our model. Second, we measure the similarity between outer-surface voxels and inner-surface voxels (denoted as 8oi), the similarity between outer-surface voxels and surface voxels (denoted as 80s) and the similarity between inner-surface voxels and surface voxels (denoted as s;g). Since there are large number of voxels in each volume, measuring

---

the similarity between all pairs of voxels is computationally impossible. Instead, we compute the feature prototypes of surface, non-surface, suner-surface and outer-surface voxels by averaging the features. Then, we measure the similarity between each class based on the cosine similarity between prototypes. We average the similarity over all test scenes and get Soi 0.6956, Sos = 0.8933 and Sie 0.8829. This also demonstrates that inner-surface voxels are quite different from outer-surface voxels. We also analyzed the accuracy of discriminating inner-surface, outer-surface and surface voxels. The results show that only few inner-surface voxels are misclassified as outer-surface voxels and only a few outer-surface voxels are misclassified as inner-surface voxels. This also supports our assumption that outer-surface voxels are quite different from inner-surface voxels since the model can easily distinguish them in most cases. For more details about this experiment, please refer to our supplementary material.
Visualization. To have an intuitive comparison with existing works, we visualize the 3D meshes reconstructed by our model and other methods in Figure. E As we can see, the reconstructed meshes of our model are more precise and complete than existing methods. Taking the first scene as an example, our model can precisely reconstruct flat desktop while the desktop reconstructed by Atlas and NeuralRecon are uneven. And compared with VoRTX, we can reconstruct the table leg which is lost in their result.
5 Conclusion
In this work, we propose the inner-outer aware reconstruction (IOAR) model for monocular 3D scene reconstruction. Different from existing methods, IOAR can classify the inner-surface voxels and outer-surface voxels, which contributes to precise occupancy prediction and TSDF prediction. To achieve this goal, design new inner-outer aware coarse-to-fine strategy and loss, which leads the model to learn to classify the inner-surface voxels and outer-surface voxels. In addition, to avoid the mutual interference between TSDF prediction and occupancy prediction, IOAR separates the occupancy branches from the TSDF branches. Thanks to these effective designs, IOAR achieves state-of-the-ar performance on the large-scale indoor scene dataset ScanNet and generalizes well on unseen scenes.
Acknowledgement
This work was supported partially by the NSFFC0021120077),1111100 Guangdong NSF Project (No. 2023B1 2020B 1515120885.
References
[1] C. Campos, R. Elvira, J. Rodríguez, M. M. Montiel, and D. Tardós, "ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM," IEEE Transcation Robotics, vol. 37, no. 6, pp. 1874-1890, 2021.
[2] L. von Stumberg, Usenko, and D. Cremers, "Direct sparse dynamic marginalization," in International Conference on Robotics and Automation, pp. 2510- 2517, IEEE, 2018
odometry using
[3] J. L. Schönberger and J. Frahm, "Structure-frowe moommmo revisited," in Conference Computer Vision and Pattern Recognition, pp. 4104-4113 13, IEFF, 2016.
[4] A. Düzęeker, S. Galliani, C. Vogel, P. Speciale, M. Dusmanu, and M. Pollefeys, "Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion," in Conference on Computer
Vision and Pattern Recognition, pp. 15324-15333, IEEE, 2021.
[5] K. Wang and S. Shen, "Mydepthnet: Real-time multiview depth estimation neural network," in International Conference on 3D Vision, pp. 248-257, IEEE, 2018.
[6] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, "Mysnet: Depth inference for unstructured multi- view stereo," in European Conference Computer Vision, vol. 11212, pp. 785-801, Springer, 2018.
[7] Y. Hou, Kannala, and A. Solin, "Multi-view stereo by temporal nonparametric fusion," in International Conference Computer Vision, pp. 2651-2660, IEEE, 2019.
10

---

[8] Z. Murez, T. van As. J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich, "Atlas: End-to-end 3d scene reconstruction from posed images." in European Conference on Computer Vision, vol. 12352, pp. 4144331 Springer, 2020.
[9] J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao, "Neuralrecon: Real-time coherent 3d recon- struction from monocular video," in Conference on Computer Vision and Pattern Recognition.
pp. 15598- 15607, IEEE, 2021.
[10] A. Bozic, P. R. Palafox. J. Thies, A. Dai, and M. Nießnnr. "Transformerfusion:: Monocular RGB scene reconstruction using transformers," " in Advances in Neural Information Processing Systems, pp. 1403-1414, 2021.
[11] N. Stier. A. Rich, P. Sen, and Т. Höllerer, "Vortx: Volumetric 3d reconstruction with transformers for voxelwise view selection and fusion," " International Conference on 3D Vision, pp. 320-330, IEEE, 2021.
[12] W. Yuan, X. Gu, H. Li, Z. Dong, and S. Zhu, "Sdf-former: Monocular scene reconstruction with 3d sdf transformers," " in International Conference on Learning Representations, pp. 1-17, 2023.
[13] W. E. Lorensen and H. E. Cline, "Marching cubes: A high resolution 3d surface construction algorithm," in Annual Conference on Computer Graphics (M. C. Stone, ed.). pp. 163-169, ACM, 1987.
[14] J. L. Schönberger, E. Zheng, J. Frahm, and M. Pollefeys, "Pixelwise view selection for unstruc- tured multi-view stereo," in European Conference on Computer Vision, pp. 501-518, Springer, 2016.
[15] Y. Furukawa and J. Ponce, "Accurate, dense, and robust multiview stereopsis," IEEE Transaction on Pattern Analaysis and Machine Intellgence, vol. 32, no. 8, pp. 1362-1376, 2010.
[16] Y. Furukawa and C. Hernández, "Multi-view stereo: tutorial," Foundations and Trends in Computer Graphics and Vision, vol. 9, no. 1-2. pp. 1-148, 2015.
[17] E. Zheng, E. Dunn, V. Jojic, J.d Frahm, "Patchmatch based joint view selection and depthmap estimation," Conference on Computer Vision and Puttern Recognition, pp. 1510-1517, IEEE, 2014.
[18) S. Im, H. Jeon, S. Lin, and I. S. Kweon, "Dpsnet: End-to-end deep plane sweep stereo," in International Conference on Learning Representations, 2019.
[19] M. Sayed, Gibson, J. Watson, V. Prisacariu, M. Firman, and C. Godard, "Simplerecon: 3d reconstruction without 3d convolutions," in European Conference on Computer Vision, vol. 13693, pp. 1-19, Springer, 2022.
[20] S. Cheng, Z. Xu, S. Zhu, Z. Li, L. E. Li, R. Ramamoorthi, and H. Su, "Deep stereo using adaptive thin volume representation with uncertainty awareness," in Conference on Computer Vision and Pattern Recognition, pp. 2521-2531 IEEE, 2020.
[21] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and Tan, "Cascade cost volume for high-resolution multi- view stereo and stereo matching," in Conference on Computer Vision and Pattern Recognition, pp. 2492-2501, IEEE, 2020.
[22] C. Liu, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz, "Neural rgb(r)d sensing: Depth and uncertainty from a video camera," in Conference on Computer Vision and Pattern Recognition, pp. 10986-10995, IEEE, 2019.
[23] Xie, C. Lei, Z. Li, L. E. Li, and Q. Chen, "Video depth estimation by fusing flow-to-depth proposals." in International Conference on Intelligent Robots and Systems, pp. 10100-10107, IEEE, 2020.
[24] M. M. Kazhdan, M. Bolitho, and H. Hoppe, "Poisson surface reconstruction." in Eurographics Symposium on Geometry Processing, vol. 256, pp. 61-70, Eurographics Association, 2006.
[25] M. M. Kazhdan and H. Hoppe, "Screened poisson surface reconstruction." ACM Transcations on Graphics, vol. 32, no. 3, pp. 29:1-29:133 2013.
[26] P. Labatut, J. Pons, and R. Keriven, "Robust and efficient surface reconstruction from range data," " Computer Graphics Forum, vol. 28, no. 8. pp. 2275 2290, 2009.
[27] R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, Kim, A. J. Davison, P. Kohli,
J. Shotton, S. Hodges, and A. W. Fitzgibbon, "Kinectfusion: Real-time dense surface mapping
11

---

and tracking," in International Symposium on Mixed and Augmented Reality, pp. 127-136, IEEE, 2011.
[28] M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, "Real-time 3d reconstruction at scale using voxel hashing," ACM Transcation on Graphics, vol. 32, no. 6, pp. 169:1-169:11, 2013.
[29] O. Kähler, V. A. Prisacariu, J. C. Valentin, and D. W. Murray, "Hierarchical voxel block hashing for efficient integration of depth images," IEEE Robotics Automation Letters, vol. 1,
no. 1, pp. 192-197, 2016.
[30] B. Curless and M. Levoy, "A volumetric method for building complex models from range images," in Annual Conference on Computer Graphics and Interactive Techniques, pp. 303-312,
ACM, 1996.
[31] M. Ji, J. Gall, H. Zheng. Y. Liu, and L.. Fang, "Surfacenet: An end-to-end 3d neural network for multiview stereopsis," in International Conference on Computer Vision, pp. 2326-2334, IEEE, 2017.
[32] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," in in Neural Information Processing Systems Workshop on Deep Learning, 2014.
[33] I.. M. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, "Oecupancy networks: Learning 3d reconstruction in function space," in Conference on Computer Vision and Pattern Recognition, pp. 44660-477, IEEE, 2019.
[34] J. Park, P. R. Florence, J. Straub, R. A. Newcombe, and S. Lovegrove, "Deepsdf: Learning continuous signed distance functions for shape representation," in Conference on Computer Vision and Pattern Recognition, pp. 165-174, IEEE, 2019,
[35] L. Liu, J. Gu, K. Z. Lin, T. Chua, and C. Theobalt, "Neural sparse voxel fields," in Advances in Neural Information Processing Systems, 2020.
[36] X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu, "Nerfusion: Fusing radiance fields for large-scale scene reconstruction," Conference on Computer Vision and Pattern Recognition,
pp. 5439 11399-5448, IEEE, 2022.
[37] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
[38] H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han, "Searching efficient 3d architectures with sparse point- convolution," in European Conference Computer Vision, pp. 685-702, Springer, 2020.
[39] X. Long, L. Liu, W. Li, C. Theobalt, and W. Wang. "Multi-view depth estimation using epipolar spatio-temporal networks," in Conference on Computer Vision and Puttern Recognition, pp. 8258-8267, IEEE, 2021.
[40] A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, "Scannet: Richly-annotated 3d reconstructions of indoor scenes," in Conference on Computer Vision and Pattern Recognition, pp. 2432-2443, IEEE, 2017.
[41] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, "A benchmark for the evaluation of RGB-D SLAM systems," in International Conference on Intelligett Robots and Systems, pp. 573-580, IEEE, 2012.
[42] A. Handa, T. Whelan, J. McDonald, and A. Davison, "A benchmark for RGB-D visual odom- etry, 3d reconstruction and SLAM," International Conference on Robotics and Automation, pp. 1524-1531, IEFF, 2014.
[43] D. P. Kingma and J. Ba. "Adam: A method for stochastic optimization," in International Conference on Learning Representations (Y. Bengio and LeCun, eds.), 2015.
[44] M. Tan, B. Chen, R. Pang. V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, "Mnasnet: Platform-awee aware neural architecture search for mobile," in Conference on Computer Vision and Pattern Recognition, pp. 2815-2823, 2019.
[45] T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, "Feature pyramid networks for object detection," in Conference on Computer Vision and Pottern Recognition, pp. 936-944. IEEE, 2017.
12

---

[46] L. van der Maaten and G. Hinton, "Visualizing data using t-sne," Journal of Machine Learning Research, vol. no. 86, pp. 2579-2605, 2008
13


Reviews:
[{"review_id": "UZlIaSsIxQ", "reviewer": "Reviewer_5s3L", "summary": "This paper introduces a novel method for monocular 3D scene reconstruction, termed Inner-Outer Aware Reconstruction (IOAR). In contrast to prior works, IOAR incorporates a unique classification process for outer surfaces, inner surfaces, and surface voxels. Additionally, the method distinctly separates the occupancy branches for IOAR from the TSDF branches, effectively mitigating any mutual interference between them. Notably, the proposed method exhibits exemplary performance across multiple datasets, including ScanNet, ICL-NUIM, and TUM-RGBD. Furthermore, the model, when trained solely on the ScanNet dataset, still achieves superior performance in both ICL-NUIM and TUM-RGBD datasets.", "questions": "- What is the rationale behind defining the occupancy ground truth to range from 0 to 2? Additionally, how are other values, such as those ranging from -1 to 1, handled in Equation (6)?\n- What is the performance of the model when trained on the ICL-NUIM and TUM-RGBD datasets and subsequently tested on the respective datasets?\n- It appears that the proposed method continuously learns to discern between the three distinct regions, even during inference. If this assumption is correct, how does the neural network ascertain these three different regions within the voxels from just the initial few frames? It seems plausible that there may not be sufficient occupied 3D voxels to fully comprehend the 3D scene initially. As more frames are encoded into 3D voxels, the neural network may begin to recognize the three regions. Is the information primarily extracted from the visual input or from the coarsely reconstructed 3D voxels?", "limitations": "Yes, in supplementary materials.", "rating": 5, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The central concept underlying this paper is both logically sound and remarkably straightforward, yet it proves to be highly effective. The provided ablation study reinforces the efficacy of the proposed module.\n- The IOAR method demonstrates unparalleled performance across a range of datasets, indicating its robustness and adaptability.", "weaknesses": "- While the contributions are impactful, they are rather direct and simplistic, which could potentially limit the depth and breadth of the paper.\n- There is an inconsistency between the numerical values presented in lines 250-251 on page 7 and those in Table 3. For instance, 0.596 is cited in the text, whereas 0.607 is reported in the table. Similar discrepancies are noted for 0.569 (text) vs. 0.564 (table), and 0.488 (text) vs. 0.507 (table). This discrepancy warrants clarification.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: While the contributions are impactful, they are rather direct and simplistic, which could potentially limit the depth and breadth of the paper.** \n\n**Re:** The insight behind IOAR is intuitive, but we believe it can inspire future works on 3D reconstruction. The starting point of IOAR is the fact that classifying inner-surface voxels and outer-surface voxels into the same class will force the model to waste its capability to bridge the gap between the two types of voxels. IOAR explores a new coarse-to-fine strategy to solve this problem and achieves performance improvement. The value of IOAR is more than the performance improvement. Existing works improve the performance of 3D reconstruction in various ways, such as adding a transformer mechanism to weigh the features [10, 11] and designing a stronger backbone [12]. On a higher level, they focus on improving 3D reconstruction by increasing the model capability. Different from them, IOAR provides another direction, i.e., to reduce the waste of the model capability. IOAR proves that reducing the waste of model capability is a potential way to improve 3D reconstruction with limited cost. Therefore, IOAR can inspire future works to try to enhance 3D reconstruction by improving the usage of model capability instead of increasing the model capability. We believe this is more valuable than performance improvement since the extensive computational cost is a critical problem in 3D reconstruction, and generally, reducing the waste of model capability requires less computational cost than increasing the model capability.\n\n **Q2: There is an inconsistency between the numerical values presented in lines 250-251 on page 7 and those in Table 3.**\n\n**Re:** The values presented in Table 3 are correct. We updated the new experiment result in Table 3 while forgetting to update those on page 7. We will fix this problem in the camera-ready version.\n\n**Q3: What is the rationale behind defining the occupancy ground truth to range from 0 to 2? Additionally, how are other values, such as those ranging from -1 to 1, handled in Equation (6)?**\n\n**Re:** Conventionally, the Signed Distance Function (SDF) measures the distance between each voxel and its closest surface. And, the Truncated Signed Distance Function (TSDF) truncated the distance at a given threshold (e.g., 12cm) and normalized the distance to range from -1 to 1 (the sign indicates the inner or outer side). Based on the definition of TSDF, we can easily define the inner-surface, outer-surface, and surface voxels in Equation (6). We define the ground-truth occupancy of inner-surface, surface, and outer-surface voxels as 0, 1, and 2 because assigning ground-truth labels from 0 is conventional in classification tasks. In this case, we can easily indicate the probability that our model makes the correct prediction using a subscript (e.g., in Equation (8)). Theoretically, the ground-truth occupancy of inner-surface, surface, and outer-surface voxels can be defined as any discrete numbers (e.g., -1, 0, 1). \n\n**Q4: What is the performance of the model when trained on the ICL-NUIM and TUM-RGBD datasets and subsequently tested on the respective datasets?**\n\n**Re:** Following previous methods [11,12], we evaluate ICL-NUIM and TUM-RGBD datasets with the model trained on the ScanNet dataset. The reason why previous methods and our model tested on ICL-NUIM and TUM-RGBD datasets using a model trained on ScanNet dataset is ICL-NUIM and TUM-RGBD datasets are too small. Specifically, the ICL-NUIM dataset contains only 8 scenes and the TUM-RGBD dataset contains only 13 scenes. On the contrary, the ScanNet dataset contains 1,613 scenes (1,513 for training and 100 for testing). Because of the lack of data, training on ICL-NUIM and TUM-RGBD datasets and subsequently testing on the respective datasets may result in poor performance. \n\nWe conduct experiments to verify this assumption. We manually split ICL-NUIM and TUM-RGBD datasets, with 6 and 10 scenes for training and 2 and 3 scenes for testing, respectively. The experiment result is worse than expected. With limited data for training, the model failed to learn to reconstruct any surface in the test scenes. That is, no meshes are reconstructed for the test scenes. This result explains why previous methods do not evaluate their model under this setting.\n\n**Q5: How does the neural network ascertain these three different regions within the voxels from just the initial few frames in the inference phase? Is the information primarily extracted from the visual input or from the coarsely reconstructed 3D voxels?**\n\n**Re:** In the inference phase, our model does not have to ascertain three different regions from just the initial few frames. Following previous works [8,9,10,11,12], given an input video, we first select $N$ ($N=60$ in the inference time) frames from the entire video. We follow the selection method in [9] to reduce redundancy, which ensures the distance and the angle between any two selected frames are large enough. Image features of all the selected frames will be utilized to generate the 3D feature volumes. Therefore, Our model can consider the information from all selected frames to predict occupancy (i.e., whether a voxel belongs to the inner region, outer region, or on the surface). Since images are the origin of all information, we believe the information is primarily extracted from the visual input."}, {"review_id": "ZHTG8LrdKb", "reviewer": "Reviewer_HJrK", "summary": "This paper proposes an inner-outer aware reconstruction (IOAR) model for monocular 3D scene reconstruction. Different from existing methods, IOAR can classify the inner-surface voxels and outer-surface voxels, which could lead to better occupancy prediction and TSDF prediction. More specifically it proposes a new inner-outer aware coarse-to-fine strategy and loss, which leads the model to learn to classifier the inner-surface voxels and outer-surface voxels. To avoid the mutual interference between TSDF prediction and occupancy prediction, IOAR separates the occupancy branch from the TSDF branch. Experimental results show IOAR achieves state-of-the-art performance on the large-scale indoor scene dataset ScanNet and can generalize reasonably on unseen scenes.\n", "questions": "How about other datasets?", "limitations": "the loss function of equation 9 is a sum of all losses, should it be weighted? How should the weights be set to?", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Overall the idea of further classifying exterior and interior surface voxels has merit and seems to generate good results. Separate TSDF and occupancy also make sense. The results are good.", "weaknesses": "only scannet is used for testing. would be nice to test on more.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: only ScanNet is used for testing. would be nice to test on more.**\n\n**Re:** In the paper, we have tested our model on three different datasets: ScanNet (Tables 1 and 2), ICL-NUIM (Table 3), and TUM-RGBD (Table 3). Following previous methods [11, 12], we evaluate the performance of IOAR on ICL-NUIM and TUM-RGBD datasets with a model trained on the ScanNet dataset. The reason why our model and previous methods test on ICL-NUIM and TUM-RGBD datasets in this manner is ICL-NUIM and TUM-RGBD datasets are too small. Specifically, the ICL-NUIM dataset contains only 8 scenes and the TUM-RGBD dataset contains 13 scenes. On the contrary, the ScanNet dataset contains 1,613 scenes (1,513 for training and 100 for testing). Since the lack of data, training on ICL-NUIM and TUM-RGBD datasets and subsequently testing on the respective datasets may result in poor performance. \n\nWe conduct experiments to verify this assumption. We manually split ICL-NUIM and TUM-RGBD datasets, with 6 and 10 scenes for training and 2 and 3 scenes for testing, respectively. The experiment result is worse than expected. With limited data for training, the model failed to learn to reconstruct any surface in the test scenes. That is, no meshes are reconstructed for the test scenes. This result explains why previous methods do not evaluate their model under this setting.\n\n**Q2: the loss function of equation 9 is a sum of all losses, should it be weighted? How should the weights be set to?**\n\n**Re:** Assigning different weights to different losses may result in better performance, while we simply set equal weights to each loss since these losses are in the same order of magnitude. To evaluate the impact of assigning different weights to different losses, we split the losses into three groups: the occupancy losses, the TSDF loss and the projective occupancy losses. We conduct experiments to set different weights to three groups of losses. We denote the weight for occupancy losses as $\\lambda_1$, the weight for TSDF loss as $\\lambda_2$,  and the weight for projective occupancy losses as $\\lambda_3$. So, the weighted loss $\\mathcal{L}^W = \\lambda_1(\\mathcal{L}^c_{OCC} + \\mathcal{L}^m_{OCC}) + \\lambda_2\\mathcal{L}^f_{TSDF} + \\lambda_3(\\mathcal{L}^c_{P} + \\mathcal{L}^m_{P} + \\mathcal{L}^f_{P}).$   The results are reported in following table. All variants achieves comparable performance except the one set $\\lambda_1$ to 0.5. In summary, assigning different weights to these losses can lead to slight performance variance, especially decreasing the weight of occupancy losses.\n\n\n\n| $\\lambda_1$ | $\\lambda_2$ | $\\lambda_3$ | Acc   | Comp  | Prec  | Recall | F-score |\n| ----------- | ----------- | ----------- | ----- | ----- | ----- | ------ | ------- |\n| 0.5         | 1           | 1           | 0.045 | 0.095 | 0.733 | 0.580  | 0.646   |\n| 1           | 0.5         | 1           | 0.046 | 0.087 | 0.740 | 0.600  | 0.661   |\n| 1           | 1           | 0.5         | 0.044 | 0.090 | 0.744 | 0.597  | 0.661   |\n| 1           | 1           | 1           | 0.043 | 0.090 | 0.748 | 0.597  | 0.663   |"}, {"review_id": "w1wmeLLoHw", "reviewer": "Reviewer_seEC", "summary": "This paper modifies previous coarse-to-fine frameworks to classify voxels into outer-surface, inner-surface, and surface voxels. In addition, the TSDF branch is added to further improve the performance. Extensive experiments show the good performance of the proposed method.", "questions": "See weaknesses.", "limitations": "No limitations are mentioned in this paper.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The motivation is interesting.\n2. Extensive experiments are conducted and the performance is good.", "weaknesses": "1. This paper proposes to predict TSDF and reformulate the occupancy into 3 cases. However, the basic framework is built based on 3D-Former and VoRTX (i.e., coarse-to-fine). Hence, the novelty is somehow limited.\n2. In Line 55, it would be confusing to say that \"IOAR explores a new coarse-to-fine strategy\" since the coarse-to-fine framework is similar to previous works. \n3. As a SoTA method, 3D-Former is not compared in the experiments.\n\nTypos:\n- Line 140: voRTX -> VoRTX\n- Table 2: Vortx [8] -> VoRTX [11]", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The novelty of this paper is somehow limited since their model is built based on the coarse-to-fine framework like previous methods (e.g., 3D-Former and VoRTX).**\n\n**Re:**  Using the basic coarse-to-fine framework does not mean the novelty of IOAR is limited. As we have introduced in the abstract and introduction, using the coarse-to-fine framework to reduce the cubically increasing memory cost is necessary for volumetric-based methods [8,9,10,11,12]. After Atlas [8] proposed the basic coarse-to-fine framework, succeeding methods like NeuralRecon[9], TransformFusion [10], VoRTX [11], and 3D-Former [12] are all built based on this basic framework. Although these works are based on the coarse-to-fine framework, their novelty is not limited since they improve the 3D reconstruction in different aspects. So does the proposed IOAR. To make it more specific, we briefly introduce the novelty of IOAR and the mentioned VoRTX and 3D-Former. VoRTX weights the contribution of image features from different views for each voxel to extract more informative feature volumes. 3D-Former design a 3D Transformer to replace the 3D CNN, which provide a stronger 3D backbone for 3D reconstruction. Different from them, IOAR aims to classify the potential surface voxels more accurately with a new coarse-to-fine strategy, which classifies the inner-surface, outer-surface, and surface voxels into different classes. Since VoRTX, 3D-Former, and IOAR improve the 3D reconstruction in different aspects, they are three complementary and orthogonal works with their own novelty. \n\n**Q2: In Line 55, it would be confusing to say that \"IOAR explores a new coarse-to-fine strategy\" since the coarse-to-fine framework is similar to previous works.**\n\n**Re:** IOAR does explore a new coarse-to-fine strategy. As we have explained in Q1, all volumetric-based methods [8,9,10,11,12] are built based on the basic coarse-to-fine framework, which utilizes a coarse-to-fine strategy to decide which voxels will keep at the next stage. The conventional coarse-to-fine strategy distinguishes potential surface voxels from other voxels and reduces the memory cost by removing the voxels far from the surface. However, this strategy groups the inner-surface voxels and outer-surface voxels into the same class, which neglects the fact that the inner-surface voxels are different from the outer-surface voxels. As a result, the classifier has to spend its capability to bridge this intrinsic gap which can lead to overfitting and thus harm the model generalization. To avoid this problem, IOAR explores a new coarse-to-fine strategy that classifies inner-surface, outer-surface, and surface voxels into different classes. Due to the intrinsic gap, the classifier can easily find the classification boundary between the inner-surface voxels and the outer-surface voxels. Therefore, the classifier can focus on finding the classification boundary between the surface voxels and inner-surface voxels and the classification boundary between the surface voxels and outer-surface voxels. Thanks to the merits of the new coarse-to-fine strategy, IOAR can predict surface voxels more accurately. Our experiments support this claim with both quantitive and qualitative results. \n\n**Q3: As a SoTA method, 3D-Former is not compared in the experiments.**\n\n**Re:** As we have mentioned in the related work section, 3D-Former [12] is an excellent work that designs a 3D Transformer to replace the 3D CNN used by previous works. In their paper, they conducted the ablation study to demonstrate that 3D-Former is a stronger 3D backbone compared with traditional 3D CNN. Therefore, replacing the 3D CNN with the 3D-Former can theoretically improve the performance of any existing volumetric-based 3D reconstruction methods. This is intuitive, just like replacing LeNet with ResNet can theoretically improve the performance of any downstream task. However, all previous methods use 3D CNN as the 3D backbone. Therefore, to have a fair comparison with existing works, we still use the 3D CNN as the 3D backbone. And, since we treat 3D-Former as a stronger backbone, we have not listed it in the experiment tables. An ideal way to make the result more complete is to split the experiment tables into two parts to report the results using the 3D CNN backbone and the results using the 3D-Former backbone. However, reproducing all existing methods using the 3D-Former as the 3D backbone is too computationally expensive. Specifically, training a basic 3D-Former requires 8 V100 GPUs to train 5,000 epochs (costs about 6000 GPU hours), let alone combining it with other methods.\n\n**Q4: There are two typos in the paper.**\n\n**Re:** We will fix these two typos in the camera-ready version."}, {"review_id": "qv7Ca6MUvD", "reviewer": "Reviewer_xj9n", "summary": "The paper presents a multi-resolution method for volumetric 3D reconstruction from an input video of a static scene captured by a moving camera. Camera poses must be externally provided. The key observation is that non-surface voxels inside and outside the surfaces have different properties which have been ignored by previous methods for estimating occupancy or the signed distance function. Based on this observation, the paper proposes a three-label classification of space: inner, surface and outer voxels. The claim is that learning decision boundaries between these three types is an easier problem than learning to discriminate between surface and non-surface voxels. This claim is supported by experimental results on widely used data, in comparison with SOTA methods, and by analysis of the classifier.\n\n** Post discussion update**\n\nAs I wrote during the discussion, I have been convinced by the other reviews and the authors responses to upgrade my rating.", "questions": "The distinction between inner and outer voxels is worthy of acceptance, but there are open questions. My preliminary ranking is low, but can be improved if there are satisfactory answers to the first three weaknesses pointed out above.", "limitations": "There is a paragraph in the supplement describing three challenging cases for the proposed algorithm. A sentence summarizing difficulties due to occlusion and transparency could have been included in the main paper. There is no potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "S1. The largest strength of the paper is that the argument that inner and outer voxels with respect to the surface are different makes sense, is novel, and leads to superior quantitative results. My impression is that this distinction is stronger near the surfaces, where it is most useful. I would argue that inner and outer voxels far from the surfaces are just photo-inconsistent, as we used to say, but the critical decisions have to be made near the surface.\n\nS2. Experiments are conducted on large, popular datasets, with carefully analyzed protocols, and the selection of baseline methods is satisfactory. Experimental results demonstrate that inner-outer distinction is powerful, as is the multi-resolution scheme. IOAR, the proposed method, outperforms the baselines by a wide margin on all metrics and all datasets.\n\nS3. Good generalization performance is also shown. This is an important property to enable future deployment of learning-based systems.", "weaknesses": "W1. There is a claim that occupancy and TSDF branches interfere with each other. In many other papers, it has been shown that multi-task learning is beneficial when the network learns related tasks. Here, there is a common 3D backbone that generates input for the occupancy and TSDF heads, so it is possible that synergies do exist and the point I am raising is due to lack of clarity. The corresponding ablation study, summarized in Table 4, does not shed enough light on how separate the TSDF and occupancy branches should be. Does the absence of branches mean that occupancy is predicted from the TSDF or vice versa? In any case, the differences are small.\n\nW2. Equation (9), the loss, requires further explanation. Why are occupancy losses applied to the coarse and medium resolution and the TSDF loss to the fine resolution? I assume that this combination worked best in practice, but I would like to see the intuition behind it. Hopefully, the reasons/observations that led to this choice could be useful to other researchers.\n\nW3. Despite criticism of volumetric methods in the abstract, the proposed approach is fully volumetric. Standard techniques to reduce the memory footprint, such as octrees and hashing, are applicable similarly to most other volumetric methods. The criticism in line 4 is unwarranted. (It is also not the best way to introduce the contribution of IOAR.)\n\nW4. Some additional ablation studies would be informative. If the authors already have the relevant data even on parts of the datasets, I would like to see (i) the effects of the multi-resolution scheme compared to a single-resolution implementation, and (ii) the accuracy in discriminating inner vs outer voxels.\n\nW5 (minor). Lines 195-196 contain: “At the fine level, all voxels are treated as potential surface voxels. Therefore, we do not set a fine-level occupancy branch.” This is unclear to me. Conceptually, there should be more empty voxels at a finer resolution since surfaces have zero thickness.\n\nW6 (minor). Lines 286-291 describe an unsatisfactory procedure for measuring the similarity across the three voxel types. Assuming that the distributions are multi-modal, which is likely, representing each category by the average may be a poor approximation. Randomly sampling thousands of representatives from each category would have been better.\n\n\nMinor Comments\n\nThe survey of related work is thorough. I have two suggestions, but their degree of overlap with the proposed method is similar to other methods.\n\nLiu C, Gu J, Kim K, Narasimhan SG, Kautz J. Neural RGB->D sensing: Depth and uncertainty from a video camera. CVPR 2019.\n\nXie J, Lei C, Li Z, Li LE, Chen Q. Video depth estimation by fusing flow-to-depth proposals. IROS 2020.\n\n\n34, 117: “marching cubes”\n\n41: “their observations”\n\n42: “there is” and “an intrinsic” – “intrinsical” is not a word and it appears several times in the paper.\n\n58, 198, among other instances: “inter-outer” should be corrected.\n\n86: “a 3D U-Net predicts”\n\n98-105: I disagree that Occupancy Networks (please correct spelling) and DeepSDF focus at novel view synthesis rather than geometric accuracy.\n\nEqs. (1), (3), (4), (5), (7): large spaces between the two parts of these equations would help.\n\n176: “state” would be better than “situation” here.\n\n179: “operations… are”\n\n193: “the rest procedure” should be corrected.\n\nMany occurrences: there should be a space between a word and a reference in square brackets.\n\nTable 3: the reference to VoRTX is wrong.\n\nSupp. 42: “Broader”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The difference between using separate or shared TSDF and occupancy branches is not clear.**\n\n**Re:** To clarify the difference, we explain how existing methods and IOAR predict occupancy and TSDF. Conventionally, a shared 3D CNN is used to refine the 3D feature volume. Then the TSDF and occupancy heads (i.e., two MLPs) predict the TSDF and occupancy for each voxel based on the same refined 3D feature volume. We notice that it is difficult for a 3D CNN to learn such versatile features that can provide information for both TSDF and occupancy prediction. Therefore, different from existing methods, IOAR contains two separate 3D CNNs for TSDF and occupancy prediction. We experimentally demonstrate the effectiveness of this design in the ablation study (section 4.2). The full IOAR model with separate 3D CNNs for TSDF and occupancy heads (line 4 of Table 4) outperforms the IOAR variant using a shared 3D CNN for TSDF and occupancy heads (line 3 of Table 4).\n\n**Q2: Why are occupancy losses applied to the coarse and medium resolutions and the TSDF loss to the fine resolution?**\n\n**Re:** The occupancy prediction aims to reduce memory cost by reducing the voxel number in the next stage. Since we filter out non-surface voxels at the end of coarse and medium levels, we only supervise the occupancy prediction in these two resolutions. Similarly, we aim to reconstruct the surface in fine resolution by predicting the TSDF. So, we only supervise the TSDF prediction in fine resolution. We conduct experiments to evaluate different variants. Here, \"IOAR\" is the original IOAR model; \"IOAR all Occ\" is a variant with occupancy losses at all resolutions; \"IOAR all TSDF\" is a variant with TSDF losses at all resolutions; \"IOAR all\" is a variant with occupancy and TSDF losses at all resolutions. All variants achieve comparable performance except the variant with all TSDF losses drops a bit on Recall.\n\n|Model|Acc|Comp|Prec|Recall|F-score|\n|-|-|-|-|-|-|\n|IOAR|0.043|0.090|0.748|0.597|0.663|\n|IOAR all Occ|0.044|0.093|0.750|0.592|0.660|\n|IOAR all TSDF|0.042|0.100|0.751|0.579|0.652|\n|IOAR all|0.044|0.089|0.743|0.593|0.658|\n\n**Q3: The criticism in line 4 is unwarranted.**\n\n**Re:** Line 4 does not aim to criticize the memory cost of all volumetric-based methods increases cubically. As one of the volumetric-based methods, the memory cost of IOAR also increases cubically. In fact, we aim to explain coarse-to-fine strategy is necessary for volumetric-based methods since their memory cost increase cubically. This is a misunderstanding caused by our writing. We believe rewriting line 4 as \"The memory cost of volumetric-based methods will grow cubically as the volume size increases, so a coarse-to-fine strategy is necessary for saving memory.\" can clear the misunderstanding.\n\n**Q4: I would like to see (i) a single-resolution implementation, and (ii) the accuracy of discriminating voxels.**\n\n**Re:**  For (i), a single-resolution implementation will lead to out-of-memory since the coarse-to-fine framework is necessary for volumetric-based methods [8,9,10,11,12] to reduce memory costs. As an alternative, we design an IOAR variant with only two levels. Without the medium resolution, the performance drops a lot.\n\n|Model|Acc|Comp|Prec|Recall|F-score|\n|-|-|-|-|-|-|\n|two level|0.058|0.096|0.670|0.553|0.604|\n|three level|0.043|0.090|0.748|0.597|0.663|\n\nFor (ii), we conduct experiments to evaluate the accuracy of discriminating voxels. At the coarse level, only 2.67% and 4.46% of inner and outer voxels are misclassified as outer and inner voxels. This supports our assumption that outer voxels are quite different from inner voxels since the model can easily distinguish them in most cases. At the medium level, we can observe a lot of inner and outer voxels are predicted as surface voxels. After filtering at the coarse level, most remaining voxels are close to the surface, so their features are similar to surface voxels. As a result, our model misclassifies them into surface voxels.\n\n|Coarse|Inner|Outer|Surface|\n|-|-|-|-|\n|Predict as Inner|71.83%|4.46%|4.17%|\n|Predict as Outer|2.67%|60.17%|0.77%|\n|Predict as Surface|25.50%|35.37%|95.06%|\n\n|Medium|Inner|Outer|Surface|\n|-|-|-|-|\n|Predict as Inner|47.30%|4.73%|6.14%|\n|Predict as Outer|2.48%|48.40%|4.82%|\n|Predict as Surface|50.22%|46.87%|89.04%|\n\n**Q5: Why do not authors set a fine-level occupancy branch to filter out more non-surface voxels?**\n\n**Re:** First, the occupancy prediction at the coarse and medium levels aims to reduce the memory cost at the next level by reducing the voxel number. Since there are no finer levels, we do not set a fine-level occupancy branch. Second, as demonstrated in Q4, these voxels at the fine level are very close to the surface. Therefore, adding another occupancy branch can hardly further filter out non-surface voxels. Finally, the area that does not contain a surface can be filtered out based on the predicted TSDF. Intuitively, if the TSDF value of eight voxels has the same sign, the cubic space with these voxels as its vertices does not contain a surface. We refer the reviewer to the marching cube algorithm [13] for more details.\n\n**Q6: Randomly sampling thousands of representatives from each category is better for measuring the similarity.**\n\n**Re:** We conducted experiments following the advice. In this setting, the similarity between inner and outer voxels is 0.4512, the similarity between inner and surface voxels is 0.5925, and the similarity between outer and surface voxels is 0.5441. These results also support that inner voxels are quite different from outer voxels.\n\n**Q7: There are two works related to your work.**\n\n**Re:** NeuralRGBD and DeepV2D estimate depth for an RGB video, thus related to the depth-based 3D reconstruction methods. We will add the introduction for these works in the related work section.\n\n**Q8: There are some typos in the paper.**\n\n**Re:**  We will fix these typos in the camera-ready version."}, {"review_id": "GGUD6vJS9Y", "reviewer": "Reviewer_Ga1p", "summary": "The authors propose a method for 3D reconstruction given images and the respective camera poses. Existing methods predict a TSDF volume and convert to 3D mesh using marching cubes. However, the authors proposed method as a coarse to fine strategy which uses a classifier to classify a voxel into surface, inner-surface or outer-surface voxel. Doing this they achieved better performance on 3D reconstruction", "questions": "- Perhaps section 4.3 could be written better? \n- Could you provide details on the inference time etc.,?", "limitations": "The method is subject to limitations by occlusion and partial scans along with transparent surfaces as listed in Section B (supplementary)", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The main strength of the paper is the proposed coarse-to-fine strategy which aids in the reduction of the memory costs of reconstruction and better performance on the reconstructed 3D meshes.\n- The method has been evaluated on reconstruction datasets such as ScanNet, ICL-NUIM, and TUM-RGBD \n- Code has been provided for reference. (I did not run it but have referred it when referring to the write up)\n- The related work has referenced all the relevant works in the area.", "weaknesses": "- The impact of classifying the difference between inner surface voxels and outer surface voxel is not entirely clear. Especially since the claim is that the classification of vowels into different classes brings major gains. Section 4.3 provides write up on this but it is not entirely clear from the write up cosine similarity between the average of features of voxels is providing demonstrable differences. \n\nPlease address the typos : \n1. Line 86 predicts* \n2. Line 89 merges*\n3. Line 177 get an*\n3. Line 183 medium/fine* \n4. Line 189 extracting* \n5. Line 193 rest of the procedure* \n6. Line 286 Since there are*\n7. Line 304 What is more -> In addition, \n\nThere are further typos in the supplementary section which I have not listed here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1: The impact of classifying the difference between inner surface voxels and outer surface voxels is not entirely clear.**\n\n**Re:** We showed the impact of classifying the difference between inner surface voxels and outer surface voxels with qualitative experiment results in the ablation study (Section 4.2). The experiment results in Table 4 showed that IOAR (line 4 of Table 4) outperforms the IOAR variant that groups the inner surface voxels and outer surface voxels into the same class (line 2 of Table 4). Specifically, the precision improves from 0.712 to 0.748, and the F-score improves from 0.654 to 0.663 when classifying the difference between inner surface voxels and outer surface voxels. This experiment result demonstrates that classifying the difference between inner surface voxels and outer surface voxels can lead to clear performance improvement.\n\n **Q2: The authors write section 4.3 to show the difference between inner surface voxels and outer surface voxels, but it is not entirely clear. Perhaps it could be written better?**\n\n**Re:** Thank you for your comment! We summarize the content in Section 4.3 in the following. If you are confused about any part, please question us for more details. In the ablation study (Section 4.2), we experimentally demonstrated the effectiveness of the new coarse-to-fine strategy, which classifies inner-surface voxels, outer-surface voxels, and surface voxels into different classes. In the introduction section, we claim this strategy works since it solves the problem that the classifier has to waste its capability to bridge the gap between inner-surface voxels and outer-surface voxels. Grouping inner-surface and outer-surface voxels into the same class will force the model to waste its capability only when there is a gap between these two types of voxels. Therefore, the first part of Section 4.3 aims to show this. We provide qualitative and quantitative results to show the difference between inner-surface voxel features and outer-surface voxel features. First, we visualize the feature space of surface, inner-surface and outer-surface voxels using t-SNE. As shown in Figure 3, inner-surface voxels (red points) are far from outer-surface voxels (blue points) in the feature space, thus demonstrating they are quite different. Second, we measure the similarity between inner-surface and outer-surface voxels, the similarity between inner-surface and surface voxels, and the similarity between outer-surface and surface voxels. The result shows that the similarity between inner-surface and outer-surface voxels is relatively lower. This result also demonstrates that inner-surface voxels are different from outer-surface voxels. In addition, we refer the reviewer to Questions 4 and 6 of reviewer xj9n. In response to Question 4 of reviewer xj9n, we conduct experiments to show the accuracy in discriminating voxels. The result shows that only 2.67% and 4.46% of inner and outer voxels are misclassified as outer and inner voxels at the coarse level. This also supports our assumption that there is an intrinsic gap between inner-surface voxels and outer-surface voxels since the model can easily distinguish them in most cases. In response to Question 6 of reviewer xj9n, we conduct another experiment to evaluate the similarity between different types of voxels. The result also supports that inner-surface voxels are different from outer-surface voxels.\n\n**Q3: Could you provide details on the inference time?**\n\n**Re:** IOAR costs 16 minutes and 18 seconds to reconstruct meshes for 100 scenes in the test set of the ScanNet dataset. On average, it costs 9.78 seconds per scene. The time of data loading, data preprocessing, and data postprocessing is included. If we only consider the inference time of the model, IOAR costs 14 minutes and 58 seconds to predict the TSDF volumes for 100 scenes. On average, it costs 8.98 seconds per scene. We also provide details of the inference phase. In the inference phase, the input is a video in which all frames have corresponding camera intrinsic and extrinsic. Based on the camera extrinsic, camera intrinsic, and image resolution, the boundary of the space captured by these images can be estimated. We split the entire space volume into sub-volumes to save memory. Since it is computationally expensive to consider all input frames, we select $N=60$ frames from the video for each sub-volume. We follow the selection method in [9] to reduce redundancy. Specifically, this selection method ensures that the distance between any two frames is far enough and the angle is large enough. These selected frames are input into the CNN backbone to extract image features at different scales. With the help of camera intrinsic and extrinsic, the image features are back-projected to form 3D feature volumes. A transformer module fuses the feature volumes from $N$ frames into a single feature volume. The fused feature volumes of sub-volumes are combined to form the fused feature volume of the global volume. The fused global feature volume is input into the TSDF branch and inner-outer aware occupancy branch to extract the TSDF feature volume and occupancy feature volume. The TSDF and occupancy feature volume at coarse and medium levels will pass to the next level to provide information from the previous scale. At coarse and medium levels, our model predicts the occupancy based on occupancy feature volume and then filters out inner-surface and outer-surface voxels based on the predicted occupancy. Finally, our model predicts the TSDF volume of the scene based on the TSDF feature volume at the fine level and then generates the 3D meshes using the marching cube algorithm[13].\n\n **Q4: There are some typos in the paper.**\n\n**Re:** We will fix these typos in the camera-ready version."}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new method for monocular 3D scene reconstruction. Multiple reviewers find the insight on exterior vs. interior surface voxels interesting and novel, while some questions requesting further explanation and empirical evidence were raised. The majority of these were addressed by rebuttal. After a careful evaluation of the reviews and rebuttal, the AC agrees with the assessment and recommends acceptance.
Decision: Accept (poster)