Title: Inner-Outer Aware Reconstruction Model for Monocular 3D Scene Reconstruction
Venue: NeurIPS 2023
Accepted: yes
Keywords: 3D reconstruction
Abstract: Monocular 3D scene reconstruction aims to reconstruct the 3D structure of scenes based on posed images. Recent volumetric-based methods directly predict the truncated signed distance function (TSDF) volume and have achieved promising results. The memory cost of volumetric-based methods will grow cubically as the volume size increases, so a coarse-to-fine strategy is necessary for saving memory. Specifically, the coarse-to-fine strategy distinguishes surface voxels from non-surface voxels, and only potential surface voxels are considered in the succeeding procedure. However, the non-surface voxels have various features, and in particular, the voxels on the inner side of the surface are quite different from those on the outer side since there exists an intrinsic gap between them. Therefore, grouping inner-surface and outer-surface voxels into the same class will force the classifier to spend its capacity to bridge the gap. By contrast, it is relatively easy for the classifier to distinguish inner-surface and outer-surface voxels due to the intrinsic gap. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. IOAR explores a new coarse-to-fine strategy to classify outer-surface, inner-surface and surface voxels. In addition, IOAR separates occupancy branches from TSDF branches to avoid mutual interference between them. Since our model can better classify the surface, outer-surface and inner-surface voxels, it can predict more precise meshes than existing methods. Experiment results on ScanNet, ICL-NUIM and TUM-RGBD datasets demonstrate the effectiveness and generalization of our model. The code is available at https://github.com/YorkQiu/InnerOuterAwareReconstruction.

Inner-Outer Aware Reconstruction Model for Monocular 3D Scene Reconstruction

Abstract

Monocular 3D scene reconstruction aims to reconstruct the 3D structure of scenes based on posed images. Recent volumetric-based methods directly predict the truncated signed distance function (TSDF) volume and have achieved promising results. The memory cost of volumetric-based methods will grow cubically as the volume size increases, so a coarse-to-fine strategy is necessary for saving memory. Specifically, the coarse-to-fine strategy distinguishes surface voxels from non-surface voxels, and only potential surface voxels are considered in the succeeding procedure. However, the non-surface voxels have various features, and in particular, the voxels on the inner side of the surface are quite different from those on the outer side since there exists an intrinsic gap between them. Therefore, grouping inner-surface and outer-surface voxels into the same class will force the classifier to spend its capacity to bridge the gap. By contrast, it is relatively easy for the classifier to distinguish inner-surface and outer-surface voxels due to the intrinsic gap. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. IOAR explores a new coarse-to-fine strategy to classify outer-surface, inner-surface and surface voxels. In addition, IOAR separates occupancy branches from TSDF branches to avoid mutual interference between them. Since our model can better classify the surface, outer-surface and inner-surface voxels, it can predict more precise meshes than existing methods. Experiment results on ScanNet, ICL-NUIM and TUM-RGBD datasets demonstrate the effectiveness and generalization of our model. The code is available at https: //github.com/YorkQiu/InnerOuterAwareReconstruction .

1 Introduction

Monocular 3D Reconstruction is a fundamental task in computer vision and is the basis for various applications like robotics, augmented/virtual reality and autonomous navigation. It aims at reconstructing the 3D structure of scenes or objects based on a sequence of monocular RGB images. Although camera poses can be estimated accurately with state-of-the-art SLAM [1, 2] or SfM systems [3], this task is still very challenging since the complexity of the environment. For example, indoor scenes can contain various objects with different textures under diverse illumination.

Researchers have studied this challenging problem for many years, and conventional methods solve this problem in a multi-view depth manner [4, 5, 6, 7]. They predict the depth map for each image and then fuse these depth maps to generate 3D meshes. Recently, some researchers [8, 9, 10, 11, 12] attempted to regress the 3D surface of the entire scene directly and have achieved promising results. They first extract image features from different views and then back-project them to form volume

A

B

C

View 1

View 2

View 3

View 4

View 1

View 2

View 3

View 4

Voxel A

Voxel B

Voxel C

outer-surface

voxels

surface

voxels

inner-surface

voxels

classification

boundary

existing methods

ours

(a)

(b)

(c)

D

E

F

G

H

D

D

D

E

G

B

B

B

F

H

I

J

I

J

features. Finally, they utilize a 3D backbone to predict a TSDF volume which can be converted to 3D mesh using the marching cubes algorithm [13]. As the volume size increases, the consumed memory will grow cubically. Therefore, existing methods apply a coarse-to-fine strategy to reduce memory costs. Specifically, they train a classifier for each level to classify whether a voxel is a surface voxel or not, and only the potential surface voxels will be selected and passed to the next level. However, the voxels on the inner side of the surface (inner-surface voxels) are quite different from the voxels on the outer side of the surface (outer-surface voxels). Due to the occlusion of the surface, the observations of inner-surface voxels are usually similar to the surface voxels. On the other hand, the outer-surface voxels are not occupied, and thus their observations from different views are surfaces of different instances. Hence, there is an intrinsic gap between the inner-surface voxels and outer-surface voxels. We give an example in Figure. 1. Figure. 1 (a) is a simple scene with only a circular object in the center of a room. And Figure. 1 (b) shows the observation of voxels A, B and C in four different views. As we can see, observations of inner-surface voxel A and outer-surface voxel C in views 2 and 4 are quite different (i.e., object surface vs. wall surface). In fact, the inner-surface voxel A is more similar to the surface voxel B since their observations under all views are different parts of the object's surface. Due to the large gap between inner-surface voxels and outer-surface voxels, it is improper to group all of them into one class, which requires the model to spend its capacity to bridge the gap. What is worse, this can lead to overfitting. As shown in Figure. 1 (c), previous methods have to learn complicated classification boundaries to classify the surface voxels and non-surface voxels.

By contrast, as there exists an intrinsic gap, classifying the inner-surface voxels and outer-surface voxels into two classes is relatively easy, meanwhile avoiding the waste of model capacity and alleviating the risk of overfitting. Inspired by this, we propose the inner-outer aware reconstruction (IOAR) model. Different from existing methods, IOAR explores a new coarse-to-fine strategy in which a classifier can classify whether a voxel is a surface, inner-surface or outer-surface voxel. In addition, we design separate TSDF and occupancy branches to avoid the mutual interference between TSDF prediction and occupancy prediction. Finally, we apply an inner-outer aware loss to supervise the inner-outer aware occupancy branch to learn to classify the inner-surface, outer-surface and surface voxels. Since it is easier to find classification boundaries between surface voxels and inner-surface voxels and between surface voxels and outer-surface voxels (Figure 1 (c)), IOAR can make more accurate predictions than previous methods. Experiments show that IOAR improves the F-score from 64.1 to 66.3 and the Precision from 70.8 to 74.8 on ScanNet. Further experiments on ICL-NUIM and TUM-RGBD datasets show that IOAR has good generalization performance.

2 Related Works

Depth-based 3D reconstruction methods reconstruct the 3D structure by two steps. First, estimate the depth of each selected video frame. Second, fuse these depths to reconstruct the 3D structure. Conventional methods [14, 15, 16, 17] typically use patch-matching based photometric consistency to estimate the depth. With the development of deep learning, many researchers paid attention to estimating accurate and dense depth maps using deep neural networks [6, 18, 5, 7, 4, 19, 20, 21]. MVSNet [6] extracts image features from multiple views and then builds a 3D cost volume based on the variance between reference view and source views. Then MVSNet predicts the depth map of the reference view based on the cost volume. MVDepthNet [5] builds the cost volume based on

Input:

𝑵

posed images

CNN

Backbone

CNN

Backbone

𝑭𝒏

𝒄

𝑭𝒏

𝒎

𝑭𝒏

𝒇

back-

project

𝑽𝒏

𝒄

𝑽𝒏

𝒎

𝑽𝒏

𝒇

Fusion

Fusion

Fusion

𝑽 𝒄

𝑽 𝒎

𝑽 𝒇

TSDF

Branch

TSDF

Branch

Inner-Outer Aware

Occupancy Branch

Inner-Outer Aware

Occupancy Branch

TSDF

Branch

TSDF

Branch

Inner-Outer Aware

Occupancy Branch

Inner-Outer Aware

Occupancy Branch

TSDF

Branch

TSDF

Branch

coarse-to-fine

sparsify

coarse-to-fine

sparsify

back-

project

back-

project

pixel intensity and applies an encoder-decoder architecture to predict depth in multiple resolutions simultaneously. GP-MVS [7] utilizes the Gaussian process to leverage information from previous latent-space encodings to predict precise depth maps. Instead of directly predicting depth maps, Neural RGB → D [22] estimates a Depth Probability Volume (DPV) for each input frame. They design a deep network based on Bayesian filtering theory, which accumulates the DPVs across frames to predict accurate depth maps. DeepV2D [23] proposes a differentiable flow-to-depth layer and estimates the depth maps based on the optical flow. DeepVideoMVS [4] is an LSTM based model, which explores the scene geometry information from previous steps. SimpleRecon [19] injects readily available metadata into the cost volume and achieves good performance without the help of 3D convolutions. After estimating the depth, methods like Poisson surface reconstruction [24, 25], Delaunay Triangulation [26] and TSDF Fusion [27, 28, 29, 30] can be used to generate 3D mesh based on inconsistent single-view depths.

Volumetric-based 3D reconstruction methods directly regress a volumetric data structure to solve the problem in an end-to-end manner. SurfaceNet [31] builds colored voxel cubes (CVC) based on images and explores 3D convolutional networks to predict a surface occupancy volume. Atlas [8] back-projects the selected 2D image features into a 3D feature volume. Taking this volume as input, a 3D U-Net predicts a TSDF volume in a coarse-to-fine manner. Each voxel in the TSDF volume describes the truncated signed distance to the nearest surface and the TSDF volume can be converted to 3D mesh using the marching cubes algorithm [13]. NeuralRecon [9] achieves real-time 3D reconstruction by reconstructing the scene in each local window and then merges the result using a GRU [32] module. Instead of averaging the contribution of each image feature, TransformerFusion [10] pays attention to the most relevant images using a transformer mechanism. VoRTX [11] also applies a transformer mechanism to aggregate image features. In addition, VoRTX weights the contribution of image features for each voxel based on the projective occupancy. 3D-Former [12] designs an 3D Transformer to replace the 3D CNN in previous works and clearly improves the performance. Theoretically, 3D-Former can be utilized as the 3D backbone of any volumetric-based 3D reconstruction method, and our method can significantly improve performance with its help. However, to have a fair comparison with existing works, we still use the 3D CNN as the 3D backbone.

Neural implicit representation of 3D structure. Some researchers [33, 34, 35, 36] aim to find a neural implicit representation of the 3D structure. Mescheder et al. [33] present OccupancyNetwork which learns a mapping from 3D coordinates and observations to the probability of occupancy. Park et al. [34] propose DeepSDF which predicts the SDF value of each point. NSVF [35] progressively sparsifies the space and finds mapping in the sparse space, thus highly reducing the computational cost. NeRFusion [36] combines the advantages of NeRF and TSDF-based fusion methods to achieve efficient large-scale reconstruction.

In this work, we propose the inner-outer aware reconstruction (IOAR) model which aims to predict meshes more precisely by finding better classification boundaries between surface voxels and innersurface voxels and between surface voxels and outer-surface voxels. IOAR explores a new coarse-tofine strategy in which the classifier can classify the inner-surface, outer-surface, and surface voxels. In addition, IOAR avoids the mutual interference between TSDF and occupancy prediction by using separate branches to predict TSDF and occupancy volume. Under the supervision of inner-outer aware loss, the occupancy branch can predict the potential surface more accurately. Extensive experiments on several datasets demonstrate the effectiveness and generalization of our IOAR.

3 Inner-Outer Aware Reconstruction

Given a set of N posed images { I n } N n =1 of a scene with corresponding camera intrinsics { K n } N n =1 and camera extrinsics { P n } N n =1 , our model aims to predict a TSDF volume V TSDF ∈ [ -1 , 1] N f X × N f Y × N f Z . This TSDF volume will further be converted to 3D mesh using the marching cubes algorithm [13]. We illustrate the pipeline of our model in Figure. 2. The posed images are input into a CNN backbone to extract image features at different scales. The feature of the image in view n at coarse/medium/fine level are indicated as F c n , F n m , F f n . Then, these features are back-projected to form feature volumes V c n , V n m , V f n and a fusion module fuses volumes from N views to generate fused feature V c , V m , V f . Finally, the fused features are input into the TSDF or occupancy branch to predict the TSDF or occupancy volume. In the medium and fine level, the TSDF/occupancy features and logits from the previous level are utilized to predict the TSDF/occupancy volume at current level.

3.1 Feature Volume Construction

The posed images are input into a 2D CNN backbone F ( · ) to extract 2D features at different scales. Using F ( · ) , we can get the image feature of I n at coarse/medium/fine level F c n , F n m , F f n = F ( I n ) . Here, F c n ∈ R C c in × H c × W c , F n m ∈ R C m in × H m × W m , F f n ∈ R C f in × H f × W f . For simplicity, we focus on a single resolution r ( r ∈ { c, m, f } ). Then these image features are back-projected to form feature volumes. Specifically, using corresponding camera intrinsic K n and camera extrinsics P n we can form the feature volume V r n ∈ R C r in × N r x × N r y × N r z :

Here, Π is the perspective mapping, : is the slice operator and N r x × N r y × N r z is the volume size in resolution r . Eq. (1) maps the voxel coordinates ( i, j, k ) in the world space to the pixel coordinates ( ˆ i, ˆ j ) in the image space and assigns the image features to the corresponding voxel features. This means that all voxels along a ray are filled with the same features of the corresponding pixel.

Then we need to fuse the feature volumes among all views. A straightforward method is averaging all feature volumes ( i.e. , V r = 1 N ∑ N n =1 V r n ). However, different views provide different information when predicting TSDF or occupancy of voxels. Therefore, assigning larger weights to views that provide information that contributes to TSDF or occupancy prediction results in better performance. Following VoRTX [11], we use a transformer to fuse the feature volumes and assign weights based on projective occupancy. First, a transformer [37] T r ( · ) is used to augment each single-view feature volume with information from other relevant views:

Here, v r n is the feature of a voxel in volume V r n ( i.e. , v r n ∈ V r n ). Then, we can predict the projective occupancy ˆ O r v ∈ R N :

Here, φ r ( · ) is an MLP and σ ( · ) is a sigmoid function. The corresponding ground-truth ( O r v ) n is equal to 1 if the voxel v at view n with resolution r is close enough to the surface. The detailed definition of O r v can be found in the supplementary material. We concatenate a zero to W r v to indicate voxel v is not close to surface under all N views and apply a softmax to compute a weight vector ˆ W r v ∈ R N +1 .

Then the weighted voxel feature of voxel v is computed by v r = ˆ W r v [ˆ v r 1 , ˆ v r 2 , · · · , ˆ v r N , glyph[vector] 0] . After weighting all voxels, we can get the fused feature volume V r ∈ R C r in × N r x × N r y × N r z .

3.2 TSDF and Inner-Outer Aware Occupancy Branch

Taking the fused feature volume V r as input, the TSDF and inner-outer aware occupancy branch predict the TSDF value and occupancy probability for each voxel. The TSDF/occupancy branch at each level consists of a 3D CNN backbone and a TSDF/occupancy head.

The procedure at the coarse level. We denote the 3D CNN backbone of the TSDF/occupancy branch as ψ c TSDF ( · ) , ψ c OCC ( · ) . The fused feature volume V c is input into each 3D CNN backbone and generates the TSDF feature F c TSDF ∈ R C c out × N c x × N c y × N c z and occupancy feature F c OCC ∈ R C c out × N c x × N c y × N c z by:

The TSDF head H c TSDF ( · ) is a 3D convolutional layer with the output channel equal to 1. Different from previous works, since we aim to classify the surface, inner-surface and outer-surface voxels, the occupancy head H c OCC ( · ) is a 3D convolutional layer with an output channel equal to 3. We can get the TSDF logits l c TSDF = ∈ R 1 × N c x × N c y × N c z and occupancy logits l c OCC ∈ R 3 × N c x × N c y × N c z at the coarse level by:

Inner-outer aware coarse-to-fine strategy. Before introducing the procedure in the medium and fine level, we first introduce the inner-outer aware coarse-to-fine strategy utilized by the inner-outer aware occupancy branch. Given the ground-truth TSDF y r TSDF ∈ R N r x × N r y × N r z , we can define surface voxels, inner-surface voxels and outer-surface voxels as follows. A voxel v is a surface voxel if it is very close to the surface ( i.e. , this voxel is in the truncated range of a surface ( -1 < ( y r TSDF ) v < 1 )). A voxel v is an inner-surface voxel if it is on the inner side of the surface ( i.e. , this voxel is not in the truncated range of any surface and its truncated distance is 1 ( ( y r TSDF ) v = 1 )). A voxel v is an outer-surface voxel if it is on the outer side of the surface ( i.e. , this voxel is not in the truncated range of any surface and its truncated distance is -1 ( ( y r TSDF ) v = -1 )). Then based on the definition of surface voxels, inner-surface voxels and outer-surface voxels, the occupancy ground-truth y r OCC ∈ R N r x × N r y × N r z can be defined as:

Given the occupancy probability at the coarse level p c OCC = softmax ( l c OCC ) , we can get the occupancy state at coarse level s c OCC = arg max( p c OCC ) ∈ { 0 , 1 , 2 } N c x × N c y × N c z . Intuitively, we can get an occupancy mask mask c at coarse level where mask c v = I (( s c OCC ) v = 1) . Here, I ( c ) equals to 1 if condition c is satisfied. With the help of the occupancy mask, we can sparsify the dense volume and reduce memory and computation costs. All sparse operations in our work are based on torchsparse [38]. Different from existing methods, our coarse-to-fine strategy makes it easier for the classifier to find better classification boundaries which leads to precise mesh prediction. For simplicity, we still use dense volumes in the following description.

The procedure at the medium/fine level. Different from the coarse level, the features from the previous level are utilized for better prediction in the medium and fine levels. To utilize the features in the previous level, we expand TSDF/occupancy features and logits in the previous level to have the same size as the current level. The expanded TSDF features ˆ F c TSDF ∈ R C c out × N x m × N y m × N z m and logits ˆ l c TSDF ∈ R 1 × N x m × N y m × N z m are concatenated with fused feature volume V m at the current level (medium level) to form medium level aggregated TSDF feature volume V m TSDF ∈ R ( C c out + C m in +1) × N x m × N y m × N z m which is used for extracting medium level TSDF features F m TSDF ∈ R C m out × N x m × N y m × N z m :

Here, N x m = 2 N c x , N y m = 2 N c y , N z m = 2 N c z . The procedure of generating TSDF logits l m TSDF is the same as coarse level using TSDF head H m TSDF ( · ) . Similarly, we can extract the medium-level

occupancy feature F m OCC ∈ R C m out × N x m × N y m × N z m and the rest of the procedure is the same as the coarse level. A detailed description of the expansion operation is introduced in the supplementary material.

At the fine level, all voxels are treated as potential surface voxels. Therefore, we do not set a fine-level occupancy branch. Except for this, the procedure of the fine level is the same as that of the medium level.

3.3 Training Loss

The loss function can be divided into three parts, the projective occupancy loss, the inner-outer aware occupancy loss, and the TSDF loss. Following previous methods, the projective occupancy loss L r P is a binary cross-entropy loss [11] and the TSDF loss L r TSDF is a l 1 distance loss [8]. The inner-outer aware occupancy loss can be computed by:

Here, p r OCC v is the occupancy probability of voxel v at resolution r , y r OCC v is the ground-truth occupancy label of voxel v at resolution r . By summing all losses, the total loss is:

4 Experiments

Datasets. Following previous works [8, 9, 11], we trained models on the ScanNet [40] dataset. ScanNet is a large-scale indoor scene dataset, which consists of 1613 RGB-D videos of 806 scenes. We follow the official train/eval/test split, where 1201 videos are used for training, 312 videos are used for evaluating and 100 videos are used for testing. Previous works evaluate the generalization performance of the models trained on ScanNet on TUM-RGBD [41] and ICL-NUIM [42] datasets. TUM-RGBD and ICL-NUIM datasets are also composed of RGB-D videos. Following previous works [11], 13 scenes of TUM-RGBD and 8 scenes of ICL-NUIM are used.

Evaluation metrics and protocol. Following conventional settings [9, 8, 11], we evaluate the performance using both 3D reconstruction metrics ( e.g. , Precision, Recall and F-Score) and 2D depth metrics ( e.g. , RMSE, Comp and Abs Rel). The detailed definition can be found in the supplementary material. Conventionally, Precision, Recall and F-Score are the three most important metrics. There

are two different evaluation protocols. The original protocol directly compared the predicted 3D meshes with the ground-truth 3D meshes. Bozic et al. [10] notice that a more complete reconstruction can be penalized since some part of the ground-truth 3D meshes is incomplete. To avoid this problem, they applied an occupancy mask when evaluating models. In the following sections, we will conduct experiments to evaluate our framework under both protocols to demonstrate its effectiveness.

Training Details. We use the Adam optimizer [43] with β 1 = 0 . 9 , β 2 = 0 . 999 and glyph[epsilon1] = 10 -8 . The learning rate is set to α = 10 -3 and is linearly warmed up from 10 -10 over 2000 steps. We trained our model for 500 epochs. We use the MnasNet-B1 [44] as the CNN backbone. We also apply the feature pyramid network [45] to extract features at different scales. The CNN backbone is fixed in the first 350 epochs and is finetuned with a learning rate α = 10 -4 in the last 150 epochs. The batch size is set to 4 and drops to 2 in the finetuning stage. Following previous methods [9, 11], we randomly drop out voxels to reduce the memory cost. In the training stage, we select 20 views for each video and the selection strategy is following [9]. To have a fair comparison, the voxel size of the fine/medium/coarse level is set to 4cm/8cm/16cm and the TSDF truncation distance is set to triple the voxel size. Training our model takes about 90 hours on a single Nvidia RTX 3090 graphic card.

4.1 Evaluation

To evaluate the performance of our model, we first compare the performance of our model and existing state-of-the-art methods on the large-scale indoor scene dataset ScanNet. The results are reported in Table. 1. As shown in the table, our model achieves state-of-the-art performance on almost all metrics. For example, our model improves the F-score from 0.641 to 0.663 and the Precision from 0.708 to 0.748 when evaluated based on the first protocol [11]. Based on the Second evaluation protocol [10], our model can also increase the F-score from 0.703 to 0.712 and the Precision from 0.767 to 0.791. The design of our model aims to find better classification boundaries, which can lead to precise surface voxel prediction. The significant improvement in Precision demonstrates the effectiveness of our design and shows that our model can make much more precise predictions than existing methods.

Following previous works, we also evaluate our model with 2D depth map metrics. Since volumetricbased methods directly predict TSDF volume instead of depth maps, we render the predicted 3D meshes to the image planes to get the depth maps. The results are reported in Table. 2. The results

show that our model can decrease the relative error by 14.7% from 0.061 to 0.052 and get a more accurate depth map.

To evaluate the generalizability of our model, we also conduct experiments on two other datasets ICLNUIM and TUM-RGBD. Models are trained on ScanNet and tested on ICL-NUIM and TUM-RGBD. The results are reported in Table. 3. On the ICL-NUIM dataset, the Precision is improved from 0.449 to 0.607 and the F-score is improved from 0.408 to 0.507. On the TUM-RGBD dataset, the Precision is improved from 0.280 to 0.564 and the F-score is improved from 0.229 to 0.343. Compared to existing methods, our model generalizes well on both datasets. A possible reason is that existing methods overfit the training set to find the complicated classification boundary between surface and non-surface voxels. Instead, our model finds the classification boundaries between inner-surface and outer-surface voxels, between surface and inner-surface voxels, and between surface and outer-surface voxels, which is much easier and thus alleviates overfitting.

4.2 Ablation Study

In addition to comparing our model with existing methods, we also conduct an ablation study to further explore the contribution of each part of our model. To this end, we design two variant models. The first one is using separate occupancy branches without the inner-outer aware coarse-to-fine strategy and loss. The second one uses the inner-outer aware coarse-to-fine strategy and loss but extracts occupancy and TSDF using a shared 3D CNN backbone. The results are reported in Table. 4. We separate the occupancy and TSDF branches since they may interfere with each other with a shared 3D CNN backbone. Compared to the basic model, using separate occupancy and TSDF branches can slightly improve all metrics. The TSDF prediction and occupancy prediction are both improved which demonstrates the effectiveness of this design. The inner-outer aware coarse-to-fine strategy and loss aim to better classify the surface voxels. Compared to the basic model, using inner-outer aware coarse-to-fine strategy and loss leads to significant improvement in Precision. This result shows that using the new strategy and loss can help the model make more precise predictions and better classify the surface voxels. Finally, the full IOAR model can further improve the performance, which shows that the two components can collaborate well.

4.3 Further Analysis

In this section, we further analyze our model with qualitative and quantitative results. First, we answer an important question: are inner-surface voxels really quite different from outer-surface voxels?

NeuralRecon

VoRTX

Ours

Ground-Truth

Second, we show some visualization results to compare the quality of 3D meshes reconstructed by our model and other methods.

Are inner-surface voxels really quite different from outer-surface voxels? This is an important question since grouping inner-surface and outer-surface voxels into the same class will force the model to waste its capability only when there is a gap between the inner-surface and outer-surface voxels (i.e., the inner-surface voxels are quite different from the outer-surface voxels). To answer this question, we conduct both qualitative and quantitative experiments. First, we visualize the features of surface, inner-surface, and outer-surface voxels in Figure. 3. As we can see, the features of innersurface and outer-surface voxels are quite different and the features of surface voxels usually distribute between inner-surface and outer-surface voxel features. Therefore, it is difficult for a classifier to distinguish non-surface voxels from surface voxels, which leads to a complicated classification boundary. This supports the motivation of our model. Second, we measure the similarity between outer-surface voxels and inner-surface voxels (denoted as s oi ), the similarity between outer-surface voxels and surface voxels (denoted as s os ) and the similarity between inner-surface voxels and surface voxels (denoted as s is ). Since there are a large number of voxels in each volume, measuring

the similarity between all pairs of voxels is computationally impossible. Instead, we compute the feature prototypes of surface, non-surface, inner-surface and outer-surface voxels by averaging the features. Then, we measure the similarity between each class based on the cosine similarity between prototypes. We average the similarity over all test scenes and get s oi = 0 . 6956 , s os = 0 . 8933 and s is = 0 . 8829 . This also demonstrates that inner-surface voxels are quite different from outer-surface voxels. We also analyzed the accuracy of discriminating inner-surface, outer-surface and surface voxels. The results show that only a few inner-surface voxels are misclassified as outer-surface voxels and only a few outer-surface voxels are misclassified as inner-surface voxels. This also supports our assumption that outer-surface voxels are quite different from inner-surface voxels since the model can easily distinguish them in most cases. For more details about this experiment, please refer to our supplementary material.

Visualization. To have an intuitive comparison with existing works, we visualize the 3D meshes reconstructed by our model and other methods in Figure. 4. As we can see, the reconstructed meshes of our model are more precise and complete than existing methods. Taking the first scene as an example, our model can precisely reconstruct a flat desktop while the desktop reconstructed by Atlas and NeuralRecon are uneven. And compared with VoRTX, we can reconstruct the table leg which is lost in their result.

5 Conclusion

In this work, we propose the inner-outer aware reconstruction (IOAR) model for monocular 3D scene reconstruction. Different from existing methods, IOAR can classify the inner-surface voxels and outer-surface voxels, which contributes to precise occupancy prediction and TSDF prediction. To achieve this goal, we design new inner-outer aware coarse-to-fine strategy and loss, which leads the model to learn to classify the inner-surface voxels and outer-surface voxels. In addition, to avoid the mutual interference between TSDF prediction and occupancy prediction, IOAR separates the occupancy branches from the TSDF branches. Thanks to these effective designs, IOAR achieves state-of-the-art performance on the large-scale indoor scene dataset ScanNet and generalizes well on unseen scenes.

References

C. Campos, R. Elvira, J. J. G. Rodríguez, J. M. M. Montiel, and J. D. Tardós, 'ORB-SLAM3: an accurate open-source library for visual, visual-inertial, and multimap SLAM,' IEEE Transcation Robotics , vol. 37, no. 6, pp. 1874-1890, 2021.

L. von Stumberg, V. Usenko, and D. Cremers, 'Direct sparse visual-inertial odometry using dynamic marginalization,' in International Conference on Robotics and Automation , pp. 25102517, IEEE, 2018.

J. L. Schönberger and J. Frahm, 'Structure-from-motion revisited,' in Conference on Computer Vision and Pattern Recognition , pp. 4104-4113, IEEE, 2016.

A. Düzçeker, S. Galliani, C. Vogel, P. Speciale, M. Dusmanu, and M. Pollefeys, 'Deepvideomvs: Multi-view stereo on video with recurrent spatio-temporal fusion,' in Conference on Computer Vision and Pattern Recognition , pp. 15324-15333, IEEE, 2021.

K. Wang and S. Shen, 'Mvdepthnet: Real-time multiview depth estimation neural network,' in International Conference on 3D Vision , pp. 248-257, IEEE, 2018.

Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, 'Mvsnet: Depth inference for unstructured multiview stereo,' in European Conference on Computer Vision , vol. 11212, pp. 785-801, Springer, 2018.

Y. Hou, J. Kannala, and A. Solin, 'Multi-view stereo by temporal nonparametric fusion,' in International Conference on Computer Vision , pp. 2651-2660, IEEE, 2019.

Z. Murez, T. van As, J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich, 'Atlas: End-to-end 3d scene reconstruction from posed images,' in European Conference on Computer Vision , vol. 12352, pp. 414-431, Springer, 2020.

J. Sun, Y. Xie, L. Chen, X. Zhou, and H. Bao, 'Neuralrecon: Real-time coherent 3d reconstruction from monocular video,' in Conference on Computer Vision and Pattern Recognition , pp. 15598-15607, IEEE, 2021.

A. Bozic, P. R. Palafox, J. Thies, A. Dai, and M. Nießner, 'Transformerfusion: Monocular RGB scene reconstruction using transformers,' in Advances in Neural Information Processing Systems , pp. 1403-1414, 2021.

N. Stier, A. Rich, P. Sen, and T. Höllerer, 'V ortx: V olumetric 3d reconstruction with transformers for voxelwise view selection and fusion,' in International Conference on 3D Vision , pp. 320-330, IEEE, 2021.

W. Yuan, X. Gu, H. Li, Z. Dong, and S. Zhu, 'Sdf-former: Monocular scene reconstruction with 3d sdf transformers,' in International Conference on Learning Representations , pp. 1-17, 2023.

W. E. Lorensen and H. E. Cline, 'Marching cubes: A high resolution 3d surface construction algorithm,' in Annual Conference on Computer Graphics (M. C. Stone, ed.), pp. 163-169, ACM, 1987.

J. L. Schönberger, E. Zheng, J. Frahm, and M. Pollefeys, 'Pixelwise view selection for unstructured multi-view stereo,' in European Conference on Computer Vision , pp. 501-518, Springer, 2016.

Y. Furukawa and J. Ponce, 'Accurate, dense, and robust multiview stereopsis,' IEEE Transaction on Pattern Analaysis and Machine Intellgence , vol. 32, no. 8, pp. 1362-1376, 2010.

Y. Furukawa and C. Hernández, 'Multi-view stereo: A tutorial,' Foundations and Trends in Computer Graphics and Vision , vol. 9, no. 1-2, pp. 1-148, 2015.

E. Zheng, E. Dunn, V. Jojic, and J. Frahm, 'Patchmatch based joint view selection and depthmap estimation,' in Conference on Computer Vision and Pattern Recognition , pp. 1510-1517, IEEE, 2014.

S. Im, H. Jeon, S. Lin, and I. S. Kweon, 'Dpsnet: End-to-end deep plane sweep stereo,' in International Conference on Learning Representations , 2019.

M. Sayed, J. Gibson, J. Watson, V. Prisacariu, M. Firman, and C. Godard, 'Simplerecon: 3d reconstruction without 3d convolutions,' in European Conference on Computer Vision , vol. 13693, pp. 1-19, Springer, 2022.

S. Cheng, Z. Xu, S. Zhu, Z. Li, L. E. Li, R. Ramamoorthi, and H. Su, 'Deep stereo using adaptive thin volume representation with uncertainty awareness,' in Conference on Computer Vision and Pattern Recognition , pp. 2521-2531, IEEE, 2020.

X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan, 'Cascade cost volume for high-resolution multiview stereo and stereo matching,' in Conference on Computer Vision and Pattern Recognition , pp. 2492-2501, IEEE, 2020.

C. Liu, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz, 'Neural rgb(r)d sensing: Depth and uncertainty from a video camera,' in Conference on Computer Vision and Pattern Recognition , pp. 10986-10995, IEEE, 2019.

J. Xie, C. Lei, Z. Li, L. E. Li, and Q. Chen, 'Video depth estimation by fusing flow-to-depth proposals,' in International Conference on Intelligent Robots and Systems , pp. 10100-10107, IEEE, 2020.

M. M. Kazhdan, M. Bolitho, and H. Hoppe, 'Poisson surface reconstruction,' in Eurographics Symposium on Geometry Processing , vol. 256, pp. 61-70, Eurographics Association, 2006.

M. M. Kazhdan and H. Hoppe, 'Screened poisson surface reconstruction,' ACM Transcations on Graphics , vol. 32, no. 3, pp. 29:1-29:13, 2013.

P. Labatut, J. Pons, and R. Keriven, 'Robust and efficient surface reconstruction from range data,' Computer Graphics Forum , vol. 28, no. 8, pp. 2275-2290, 2009.

R. A. Newcombe, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, P. Kohli, J. Shotton, S. Hodges, and A. W. Fitzgibbon, 'Kinectfusion: Real-time dense surface mapping

and tracking,' in International Symposium on Mixed and Augmented Reality , pp. 127-136, IEEE, 2011.

M. Nießner, M. Zollhöfer, S. Izadi, and M. Stamminger, 'Real-time 3d reconstruction at scale using voxel hashing,' ACM Transcation on Graphics , vol. 32, no. 6, pp. 169:1-169:11, 2013.

O. Kähler, V. A. Prisacariu, J. P. C. Valentin, and D. W. Murray, 'Hierarchical voxel block hashing for efficient integration of depth images,' IEEE Robotics Automation Letters , vol. 1, no. 1, pp. 192-197, 2016.

B. Curless and M. Levoy, 'A volumetric method for building complex models from range images,' in Annual Conference on Computer Graphics and Interactive Techniques , pp. 303-312, ACM, 1996.

M. Ji, J. Gall, H. Zheng, Y . Liu, and L. Fang, 'Surfacenet: An end-to-end 3d neural network for multiview stereopsis,' in International Conference on Computer Vision , pp. 2326-2334, IEEE, 2017.

J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, 'Empirical evaluation of gated recurrent neural networks on sequence modeling,' in in Neural Information Processing Systems Workshop on Deep Learning , 2014.

L. M. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger, 'Occupancy networks: Learning 3d reconstruction in function space,' in Conference on Computer Vision and Pattern Recognition , pp. 4460-4470, IEEE, 2019.

J. J. Park, P. R. Florence, J. Straub, R. A. Newcombe, and S. Lovegrove, 'Deepsdf: Learning continuous signed distance functions for shape representation,' in Conference on Computer Vision and Pattern Recognition , pp. 165-174, IEEE, 2019.

L. Liu, J. Gu, K. Z. Lin, T. Chua, and C. Theobalt, 'Neural sparse voxel fields,' in Advances in Neural Information Processing Systems , 2020.

X. Zhang, S. Bi, K. Sunkavalli, H. Su, and Z. Xu, 'Nerfusion: Fusing radiance fields for large-scale scene reconstruction,' in Conference on Computer Vision and Pattern Recognition , pp. 5439-5448, IEEE, 2022.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, 'Attention is all you need,' in Advances in Neural Information Processing Systems , pp. 5998-6008, 2017.

H. Tang, Z. Liu, S. Zhao, Y. Lin, J. Lin, H. Wang, and S. Han, 'Searching efficient 3d architectures with sparse point-voxel convolution,' in European Conference on Computer Vision , pp. 685-702, Springer, 2020.

X. Long, L. Liu, W. Li, C. Theobalt, and W. Wang, 'Multi-view depth estimation using epipolar spatio-temporal networks,' in Conference on Computer Vision and Pattern Recognition , pp. 8258-8267, IEEE, 2021.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, 'Scannet: Richly-annotated 3d reconstructions of indoor scenes,' in Conference on Computer Vision and Pattern Recognition , pp. 2432-2443, IEEE, 2017.

J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers, 'A benchmark for the evaluation of RGB-D SLAM systems,' in International Conference on Intelligent Robots and Systems , pp. 573-580, IEEE, 2012.

A. Handa, T. Whelan, J. McDonald, and A. J. Davison, 'A benchmark for RGB-D visual odometry, 3d reconstruction and SLAM,' in International Conference on Robotics and Automation , pp. 1524-1531, IEEE, 2014.

D. P. Kingma and J. Ba, 'Adam: A method for stochastic optimization,' in International Conference on Learning Representations (Y. Bengio and Y. LeCun, eds.), 2015.

M. Tan, B. Chen, R. Pang, V. Vasudevan, M. Sandler, A. Howard, and Q. V. Le, 'Mnasnet: Platform-aware neural architecture search for mobile,' in Conference on Computer Vision and Pattern Recognition , pp. 2815-2823, 2019.

T. Lin, P. Dollár, R. B. Girshick, K. He, B. Hariharan, and S. J. Belongie, 'Feature pyramid networks for object detection,' in Conference on Computer Vision and Pattern Recognition , pp. 936-944, IEEE, 2017.

L. van der Maaten and G. Hinton, 'Visualizing data using t-sne,' Journal of Machine Learning Research , vol. 9, no. 86, pp. 2579-2605, 2008.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method for 3D reconstruction given images and the respective camera poses. Existing methods predict a TSDF volume and convert to 3D mesh using marching cubes. However, the authors proposed method as a coarse to fine strategy which uses a classifier to classify a voxel into surface, inner-surface or outer-surface voxel. Doing this they achieved better performance on 3D reconstruction\n\nSTRENGTHS:\n- The main strength of the paper is the proposed coarse-to-fine strategy which aids in the reduction of the memory costs of reconstruction and better performance on the reconstructed 3D meshes.\n- The method has been evaluated on reconstruction datasets such as ScanNet, ICL-NUIM, and TUM-RGBD \n- Code has been provided for reference. (I did not run it but have referred it when referring to the write up)\n- The related work has referenced all the relevant works in the area.\n\nWEAKNESSES:\n- The impact of classifying the difference between inner surface voxels and outer surface voxel is not entirely clear. Especially since the claim is that the classification of vowels into different classes brings major gains. Section 4.3 provides write up on this but it is not entirely clear from the write up cosine similarity between the average of features of voxels is providing demonstrable differences. \n\nPlease address the typos : \n1. Line 86 predicts* \n2. Line 89 merges*\n3. Line 177 get an*\n3. Line 183 medium/fine* \n4. Line 189 extracting* \n5. Line 193 rest of the procedure* \n6. Line 286 Since there are*\n7. Line 304 What is more -> In addition, \n\nThere are further typos in the supplementary section which I have not listed here.\n\nQUESTIONS:\n- Perhaps section 4.3 could be written better? \n- Could you provide details on the inference time etc.,?\n\nLIMITATIONS:\nThe method is subject to limitations by occlusion and partial scans along with transparent surfaces as listed in Section B (supplementary)", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a multi-resolution method for volumetric 3D reconstruction from an input video of a static scene captured by a moving camera. Camera poses must be externally provided. The key observation is that non-surface voxels inside and outside the surfaces have different properties which have been ignored by previous methods for estimating occupancy or the signed distance function. Based on this observation, the paper proposes a three-label classification of space: inner, surface and outer voxels. The claim is that learning decision boundaries between these three types is an easier problem than learning to discriminate between surface and non-surface voxels. This claim is supported by experimental results on widely used data, in comparison with SOTA methods, and by analysis of the classifier.\n\n** Post discussion update**\n\nAs I wrote during the discussion, I have been convinced by the other reviews and the authors responses to upgrade my rating.\n\nSTRENGTHS:\nS1. The largest strength of the paper is that the argument that inner and outer voxels with respect to the surface are different makes sense, is novel, and leads to superior quantitative results. My impression is that this distinction is stronger near the surfaces, where it is most useful. I would argue that inner and outer voxels far from the surfaces are just photo-inconsistent, as we used to say, but the critical decisions have to be made near the surface.\n\nS2. Experiments are conducted on large, popular datasets, with carefully analyzed protocols, and the selection of baseline methods is satisfactory. Experimental results demonstrate that inner-outer distinction is powerful, as is the multi-resolution scheme. IOAR, the proposed method, outperforms the baselines by a wide margin on all metrics and all datasets.\n\nS3. Good generalization performance is also shown. This is an important property to enable future deployment of learning-based systems.\n\nWEAKNESSES:\nW1. There is a claim that occupancy and TSDF branches interfere with each other. In many other papers, it has been shown that multi-task learning is beneficial when the network learns related tasks. Here, there is a common 3D backbone that generates input for the occupancy and TSDF heads, so it is possible that synergies do exist and the point I am raising is due to lack of clarity. The corresponding ablation study, summarized in Table 4, does not shed enough light on how separate the TSDF and occupancy branches should be. Does the absence of branches mean that occupancy is predicted from the TSDF or vice versa? In any case, the differences are small.\n\nW2. Equation (9), the loss, requires further explanation. Why are occupancy losses applied to the coarse and medium resolution and the TSDF loss to the fine resolution? I assume that this combination worked best in practice, but I would like to see the intuition behind it. Hopefully, the reasons/observations that led to this choice could be useful to other researchers.\n\nW3. Despite criticism of volumetric methods in the abstract, the proposed approach is fully volumetric. Standard techniques to reduce the memory footprint, such as octrees and hashing, are applicable similarly to most other volumetric methods. The criticism in line 4 is unwarranted. (It is also not the best way to introduce the contribution of IOAR.)\n\nW4. Some additional ablation studies would be informative. If the authors already have the relevant data even on parts of the datasets, I would like to see (i) the effects of the multi-resolution scheme compared to a single-resolution implementation, and (ii) the accuracy in discriminating inner vs outer voxels.\n\nW5 (minor). Lines 195-196 contain: “At the fine level, all voxels are treated as potential surface voxels. Therefore, we do not set a fine-level occupancy branch.” This is unclear to me. Conceptually, there should be more empty voxels at a finer resolution since surfaces have zero thickness.\n\nW6 (minor). Lines 286-291 describe an unsatisfactory procedure for measuring the similarity across the three voxel types. Assuming that the distributions are multi-modal, which is likely, representing each category by the average may be a poor approximation. Randomly sampling thousands of representatives from each category would have been better.\n\n\nMinor Comments\n\nThe survey of related work is thorough. I have two suggestions, but their degree of overlap with the proposed method is similar to other methods.\n\nLiu C, Gu J, Kim K, Narasimhan SG, Kautz J. Neural RGB->D sensing: Depth and uncertainty from a video camera. CVPR 2019.\n\nXie J, Lei C, Li Z, Li LE, Chen Q. Video depth estimation by fusing flow-to-depth proposals. IROS 2020.\n\n\n34, 117: “marching cubes”\n\n41: “their observations”\n\n42: “there is” and “an intrinsic” – “intrinsical” is not a word and it appears several times in the paper.\n\n58, 198, among other instances: “inter-outer” should be corrected.\n\n86: “a 3D U-Net predicts”\n\n98-105: I disagree that Occupancy Networks (please correct spelling) and DeepSDF focus at novel view synthesis rather than geometric accuracy.\n\nEqs. (1), (3), (4), (5), (7): large spaces between the two parts of these equations would help.\n\n176: “state” would be better than “situation” here.\n\n179: “operations… are”\n\n193: “the rest procedure” should be corrected.\n\nMany occurrences: there should be a space between a word and a reference in square brackets.\n\nTable 3: the reference to VoRTX is wrong.\n\nSupp. 42: “Broader”\n\nQUESTIONS:\nThe distinction between inner and outer voxels is worthy of acceptance, but there are open questions. My preliminary ranking is low, but can be improved if there are satisfactory answers to the first three weaknesses pointed out above.\n\nLIMITATIONS:\nThere is a paragraph in the supplement describing three challenging cases for the proposed algorithm. A sentence summarizing difficulties due to occlusion and transparency could have been included in the main paper. There is no potential negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper modifies previous coarse-to-fine frameworks to classify voxels into outer-surface, inner-surface, and surface voxels. In addition, the TSDF branch is added to further improve the performance. Extensive experiments show the good performance of the proposed method.

STRENGTHS:
1. The motivation is interesting.
2. Extensive experiments are conducted and the performance is good.

WEAKNESSES:
1. This paper proposes to predict TSDF and reformulate the occupancy into 3 cases. However, the basic framework is built based on 3D-Former and VoRTX (i.e., coarse-to-fine). Hence, the novelty is somehow limited.
2. In Line 55, it would be confusing to say that "IOAR explores a new coarse-to-fine strategy" since the coarse-to-fine framework is similar to previous works. 
3. As a SoTA method, 3D-Former is not compared in the experiments.

Typos:
- Line 140: voRTX -> VoRTX
- Table 2: Vortx [8] -> VoRTX [11]

QUESTIONS:
See weaknesses.

LIMITATIONS:
No limitations are mentioned in this paper.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

comments: SUMMARY:
This paper proposes an inner-outer aware reconstruction (IOAR) model for monocular 3D scene reconstruction. Different from existing methods, IOAR can classify the inner-surface voxels and outer-surface voxels, which could lead to better occupancy prediction and TSDF prediction. More specifically it proposes a new inner-outer aware coarse-to-fine strategy and loss, which leads the model to learn to classifier the inner-surface voxels and outer-surface voxels. To avoid the mutual interference between TSDF prediction and occupancy prediction, IOAR separates the occupancy branch from the TSDF branch. Experimental results show IOAR achieves state-of-the-art performance on the large-scale indoor scene dataset ScanNet and can generalize reasonably on unseen scenes.

STRENGTHS:
Overall the idea of further classifying exterior and interior surface voxels has merit and seems to generate good results. Separate TSDF and occupancy also make sense. The results are good.

WEAKNESSES:
only scannet is used for testing. would be nice to test on more.

QUESTIONS:
How about other datasets?

LIMITATIONS:
the loss function of equation 9 is a sum of all losses, should it be weighted? How should the weights be set to?
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel method for monocular 3D scene reconstruction, termed Inner-Outer Aware Reconstruction (IOAR). In contrast to prior works, IOAR incorporates a unique classification process for outer surfaces, inner surfaces, and surface voxels. Additionally, the method distinctly separates the occupancy branches for IOAR from the TSDF branches, effectively mitigating any mutual interference between them. Notably, the proposed method exhibits exemplary performance across multiple datasets, including ScanNet, ICL-NUIM, and TUM-RGBD. Furthermore, the model, when trained solely on the ScanNet dataset, still achieves superior performance in both ICL-NUIM and TUM-RGBD datasets.\n\nSTRENGTHS:\n- The central concept underlying this paper is both logically sound and remarkably straightforward, yet it proves to be highly effective. The provided ablation study reinforces the efficacy of the proposed module.\n- The IOAR method demonstrates unparalleled performance across a range of datasets, indicating its robustness and adaptability.\n\nWEAKNESSES:\n- While the contributions are impactful, they are rather direct and simplistic, which could potentially limit the depth and breadth of the paper.\n- There is an inconsistency between the numerical values presented in lines 250-251 on page 7 and those in Table 3. For instance, 0.596 is cited in the text, whereas 0.607 is reported in the table. Similar discrepancies are noted for 0.569 (text) vs. 0.564 (table), and 0.488 (text) vs. 0.507 (table). This discrepancy warrants clarification.\n\nQUESTIONS:\n- What is the rationale behind defining the occupancy ground truth to range from 0 to 2? Additionally, how are other values, such as those ranging from -1 to 1, handled in Equation (6)?\n- What is the performance of the model when trained on the ICL-NUIM and TUM-RGBD datasets and subsequently tested on the respective datasets?\n- It appears that the proposed method continuously learns to discern between the three distinct regions, even during inference. If this assumption is correct, how does the neural network ascertain these three different regions within the voxels from just the initial few frames? It seems plausible that there may not be sufficient occupied 3D voxels to fully comprehend the 3D scene initially. As more frames are encoded into 3D voxels, the neural network may begin to recognize the three regions. Is the information primarily extracted from the visual input or from the coarsely reconstructed 3D voxels?\n\nLIMITATIONS:\nYes, in supplementary materials.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}