Monocular Scene Reconstruction with 3D SDF Transformers
Authors: Weihao Yuan
Xiaodong Gu
Heng Li
Zilong Dong
Siyu Zhu
Venue: ICLR 2023 poster (2023)
Keywords: 3D Reconstruction
Monocular Scene Reconstruction
3D Transformer
TSDF volume

Abstract: Monocular scene reconstruction from posed images is challenging due to the complexity of a large environment. Recent volumetric methods learn to directly predict the TSDF volume and have demonstrated promising results in this task. However, most methods focus on how to extract and fuse the 2D features to a 3D feature volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose an SDF transformer network, which replaces the role of 3D CNN for better 3D feature aggregation. To reduce the explosive computation complexity of the 3D multi-head attention, we propose a sparse window attention module, where the attention is only calculated between the non-empty voxels within a local window. Then a top-down-bottom-up 3D attention network is built for 3D feature aggregation, where a dilate-attention structure is proposed to prevent geometry degeneration, and two global modules are employed to equip with global receptive fields. The experiments on multiple datasets show that this 3D transformer network generates a more accurate and complete reconstruction, which outperforms previous methods by a large margin. Remarkably, the mesh accuracy is improved by 41.8%, and the mesh completeness is improved by 25.3% on the ScanNet dataset. The code of our method will be made public.

Published as a conference paper at ICLR 2023
3D FORMER: MONOCULAR SCENE RECONSTRUCTION WITH 3D SDF TRANSFORMERS
Weihao Yuan, Xiaodong Gu, Heng Li, Zilong Dong, Siyu Zhu"
Alibaba Group
{qi {qianmu. ywh, dadong gxd, bacshu. 1h, dist.dzl, siting. zsy) @a 1. bbaab--aa--a inn. com
ABSTRACT
Monocular scene reconstruction from posed images challenging due to the com- plexity of a large environment. Recent volumetric methods learn to directly pre- dict the TSDF volume and have demonstrated promising results in this task. How- ever, most methods focus on how to extract and fuse the 2D features to 3D fea- ture volume, but none of them improve the way how the 3D volume is aggregated. In this work, we propose
SDF transformer network, which replaces the role 3D CNN for better 3D feature aggregation. To reduce the explosive compu- tation complexity of the 3D multi-head attention, propose sparse window attention module, where the attention is only calculated between the non-empty voxels within a local window. Then a oop-down-bottom-ee 3D attention network is built for 3D feature aggregation, where a dilate-attention structure is proposed prevent geometry degeneration, and two global modules are employed equip with global receptive fields. The experiments 3D transformer network generates more accurate and complete reconstruction. which outperforms previous methods by large margin. Remarkably, the mesh ac- curacy improved by 41.8%, and the mesh completeness is improved by 25.3% the ScanNet dataset.
multiple datasets show that this
INTRODUCTION
Monocular 3D reconstruction is a classical task in computer vision and essential for numerous applications like autonomous navigation, robotics, and augmented/virtual reality. Such a vision task aims reconstruct an accurate and complete dense 3D shape of an unstructured scene from only sequence of monocular RGB images. While the camera poses can be estimated accurately with the stat-ooftthe-at SLAM (Campos et al., 202T) or SfM systems (Schonberger & Frahm 2016), dense 3D scene reconstruction from these posed images is still challenging problem due the complex geometry of a large-scale environment, such as the various objects, flexible lighting. reflective sur- faces, and diverse cameras of different focus, distortion, and sensor noise. Many previous methods reconstruct the scenario in multi-view depth manner (Yao et al., |2088|Chen CCen et af. 2019 Duzceker
al.) 2021). They predict the dense depth map of each target frame, which can estimate accurate local geometry but need additional efforts in fusing these depth maps (Murez et al. 2020 Sun et al., 2021
e.g., solving the inconsistencies between different views.
Recently, some methods have tried to directly regress the complete 3D surface of the entire
scene (Murez et al. 2020 Sun al. (02I)) from truncated signed distance function (TSDF)
representation. They first extract the 2D features with 2D convolutional neural networks (CNN), and then back-project the features to 3D space. Afterward, the 3D feature volume processed by 3D CNN network to output TSDF volume prediction, which is extracted to surface mesh by marching cubes (Lorensen & Cline 1987). This way of reconstruction is end-to-end trainable, and is demonstrated to output accurate, coherent, and complete meshes. In this paper, follow this volume-basee 3D reconstruction path and directly regress the TSDF volume.
"Corresponding Author Projecc Page:
thub. fo/former3d

---

Published as a conference paper at ICLR 2023
2D Feature
V(
Fusion
y2
va
OO
Former
3D Dense Volume
3D Sparse Volume
Vi
Fusion
y2
V2
3D SDF Former
0'
project
Backbone
VP
Fusion
V
3D SDF Former
S
3D
The overview of the 3D reconstruction framework. The input images are extracted to features by a 2D backbone network, then the 2D features are back-projected and fused to 3D feature volumes, which are aggregated by our 3D SDF transformer and generate the reconstruction in coarse-to-fine manner.
Figure I:
Inspired by recent successes of vision transformer VVswwnn et al,, 2017 Dosovitskiy et al. 2020 some approaches (Bozic et al. 2021; Stier et al. 2021) have adopted this structure in 3D recon- struction, but their usages are all limited to fusing the 2D features from different views while the aggregation of the 3D feature volumes is still performed by the 3D CNN. In this paper, we claim that the aggregation of 3D feature volume is also critical, and the evolution from 3D CNN to 3D multi-head attention could further improve both the accuracy and completeness of the reconstruc- tion. Obviously, the limited usage of 3D multi-head attention in 3D feature volume aggregation mainly due to its explosive computation. Specifically, the attention between each voxel and any other voxel needs to be calculated, which is hard to be realized in general computing platform. This is also the reason why there are only a few applications of 3D transformers in solving 3D tasks.
In this work, to address the above challenges and make the 3D transformer practical for 3D scene reconstruction, we propose a sparse window multi-head attention structure. Inspired by the sparse CNN (Yan et al. 2018), we first sparsify the 3D feature volume with predicted occupancy, in which way the number of the voxels is reduced to only the occupied ones. Then, to compute the attention score of a target voxel, we define local window centered on this voxel, within which the non-empty voxels are considered for attention computing. In this way, the computation complexity of the 3D be reduced by orders of magnitude, and this module be embedded into a network for 3D feature aggregation. Therefore, with this module, we build the first 3D transformer based top-down- -ootoooootttoouppppppp network, where dilate-attention module and its inverse are used to downsample and upsample the 3D feature volume. In addition, to make up for the local receptive field of the sparse window attention, we add a global attention module and a global context module at the bottom of this network since the size of the volume is very small at the bottom level. With this network, the 3D shape is estimated in a coarse-to-fine manner of three levels, as is displayed in Figure | To the best of our knowledge, this is the first paper employing the 3D transformer for 3D scene reconstruction from a TSDF representation.
multi-head attention
In the experiments, our method is demonstrated to outperform previous methods by significant margin on multiple datasets. Specifically, the accuracy metric of the mesh on the ScanNet dataset is reduced by 41.8%. from 0.055 to 0.032, and the completeness metric is reduced by 25.3%, from 0.083 to 0.062. In the qualitative results, the meshes reconstructed by our method are dense, accu- rate, and complete. The main contributions of this work are then summarized as follows:
We propose a sparse window multi head attention module, with which the computation complexity of the 3D transformer is reduced significantly and becomes feasible.
We propose a dilate-attention structure to avoid geometry degeneration in downsampling, with which we build the first sal-down-hotto---- 3D transformer network for 3D feature aggregation. This network is further improved with bottom-level global attention and global context encoding.
This 3D transformer is employed to aggregate the 3D features back-projected from the 2D features of an image sequence in a coarse-to-fine manner, and predict TSDF values for accurate and complete 3D reconstruction. This framework shows significant improvement in multiple datasets.
2 RELATED WORK
Depth-based 3D Reconstruction. In traditional methods, reconstructing a 3D model of a scene usually involves depth estimating for a series of images, and then fusing these depths together into
2

---

Published as a conference paper at ICLR 2023
3D data structure (Schönberger al. 2016). After the rising of deep learning, many works have tried to estimate accurate and dense depth maps with deep neural networks Yaoo et al.
2018 Wang & Shen 2018: Chen et al. 2019; Im et al. 2019: Yuan et al. 2021 2022 Long al.,
2021 They
usually estimate the depth map of the reference image by constructing a 3D cost volume from sev- eral frames in local window. Also, leverage the information in the image sequence, some other methods try to propagate the message from previously predicted depths utilizing probabilistic filter- ing (Liu et al. 2019), Gaussian process (Hou al.) (019a), or recurrent neural networks (Duzceker et al. (0021) Although the predicted depth maps are increasingly accurate, there is still a gap be- tween these single-view depths and the complete 3D shape. Post mesh generation like Poisson reconstruction (Kazhdan & Hoppe, 2013), Delaunay triagulation (Labatut et al., 2009), and TSDF fusion (Newcombe et al. 1011]) are proposed to solve this problem, but the inconsistency between different views is still a challenge.
Volume-based 3D Reconstruction. To avoid the depth estimation and fusion in 3D reconstruction, some methods try to directly regress a volumetric data structure end-to-end. SurfaceNet (Ji et al. 2017) encodes the camera parameters together with the images to predict a 3D surface occupancy volume with 3D convolutional networks. Afterward, Atlas (Murez al. 2020) back-projects the 2D features of all images into a 3D feature volume with the estimated camera poses, and then feeds this 3D volume into 3D U-Net to predict TSDF volume. Then NeuralRecon (Sun et al., 2021) improves the efficiency by doing this within a local window and then fusing the prediction together using a GRU module. Recently, to improve the accuracy of the reconstruction, some methods also introduce transformers to do the fusion of 2D features from different views (Bozic et al. 202 Stier al. 202T). However, their transformers are all limited in 2D space and used to process 2D features, which is not straightforward in the 3D reconstruction task.
There are also some methods for object 3D shape prediction, which can infer the 3D shape of objects with only few views (Xie et al.] [2200] Wang et al., 2021a). But the network of these methods can only infer the shape of one category of small objects. Lately, some works represent the 3D shape with an implicit network, and optimize the implicit representation by neural rendering (Yariv et al.) 2020 Wang et al. [022lb] Yariv et al. 2021). These methods could obtain a fine surface of an object with iterative optimization, but with the cost of long-timo reconstruction.
Transformers 3D Vision. The transformer structure (Vaswani et al., 2017) has attracted a lot of attention and achieved many successes in vision tasks (Dosovitskiy et al. 2020 Liu et al. 2021). Most of them, nevertheless, are used for 2D feature extraction and aggregation. Even in 2D feature processing, the computation complexity is already quite high, so many works proposed to reduce the resource-consuming (Dosovitskiy et al. 2020 Liu et al. 2021). Directly extending the trans- former from 2D 3D would cause catastrophic computation. Thus most works only carefully performed on resource-saving feature extraction, e.g., the one-off straightforward feature mapping without any downsampling or upsampling (Wang et al. 2021a), where the size of the feature vol- ume remains unchanged, or the top-down tasks with only downsampling (Mao et al. 202T), where the size of the feature volume is reduced gradually. In 3D reconstruction, however, a top-down- bottom-up structure is more reasonable for feature extraction and shape generation, as in most of the 3D-CNN-based structures (Murez et al., 20200 (unn et al., 202T Stier et al. 202T). So this work, we design the first 3D transformer based top-down-botto--ee of 3D reconstruction. In addition, a sparse window multi- head attention mechanism is proposed to save the computation cost. Although the sparse structure can handle the highly-sparss data, like the object detection of Lidar points (Mao al. (022). J. it is not suitable for processing a relatively-denss data, like mesh of an indoor scene. Therefore, sparse window structure is needed in 3D scene reconstruction, where a dense surface within a window could be sufficiently aggregated.
structure for improving the quality
METHOD
3.1 OVERVIEW
The overview framework of our method is illustrated in Figure 1 Given a sequence of images [ of a scene and the corresponding camera intrinsics j and extrinsics {P.) we first extract the image features (F in 2D space in three levels, and then back project these 2D features to 3D space, which are fused to three feature volumes in the coarse, medium, and fine levels, respectively. Afterward, these three feature volumes are aggregated by our SDF 3D transformer in
3

---

Published as a conference paper at ICLR 2023
Dilated voxel
Empty
Sparse volume
Occupied voxel
(a) Sparse Window Attention
(b) Dilate Attention
Figure 2: (a) Illustration of the sparse window attention. For calculating the attention of the current voxel (in orange), first sparsify the volume using the occupancy prediction from the coarser level, and then search the occupied voxels (in dark blue) within a small window. The attention is hence computed based on only these neighbor occupied voxels. (b) Illustration of the dilate-attention in a 2D slice. We dilate the occupied voxels and calculate the attention of these dilated voxels (in yellow) to maintain the geometry structure.
a coarse-to-fine manner. At the coarse and medium levels, the output of the 3D transformer is two occupancy volumes O O while at the fine level, the output is the predicted TSDF volume S The coarse occupancy volume O and the medium occupancy volume O store the occupancy values E [0, of the voxels, which are used sparsify the finer level. Therefore, the feature volumes could be processed sparsely to reduce the computation complexity. Finally, the predicted mesh is extracted using marching cubes (Lorensen Cline 1987) from the TSDF volume S
3.2 FEATURE VOLUME CONSTRUCTION
The 2D features (F in three levels 1 = 0, 2 are extracted by feature pyramid network (Lin jet al. (017)) with the MnasNet-BI (Tan et al. 2019) as the backbone. The resolution the features at these three levels are 3: 1
T6 respectively. Then following Murez et al. (2020), we back project
the 2D features 3D space with the camera parameters (K,) and [P: i=1 generating 3D>
feature volumes [V of size Nx Ny X Nz.
In previous work, usually the fusion of these feature volumes from different views computed by taking the average (Murez et al.) 2020 Sun et al., 2021). However, the back- croj--rojected features from different views contribute differently to the 3D shape, e.g.. the view with a bad viewing angle and the voxels far from the surface. Therefore, weighted average is more reasonable than taking the average. To compute these weights, for each voxel we calculate the variance the features of different views by
Var -V! -
(1)
where v' is the average of the features of all views. Then we feed the features and the variance into small MLP to calculate the weights W., which are used to compute a weighted average of the features from different views as
V DVV (artMaxxWW))..
where x denotes element-wise multiplication.
Inspired by Yao al. (2018), we also calculate the total variance of all feature volumes and then concatenate it with the weighted average to the final feature volumes, as
v (v! Σ Var!), N
(3)
3.3
SPARSE WINDOW MULTI-HEAD ATTENTION
The multi-head attention structure has been shown be effective in many vision tasks (Dosovitskiy jet al. 2020; Liu et al. 2021). Most of them, however, are limited to 2D feature processing rather than 3D feature processing. This because the computation complexity of the multi-head attention is usually higher than convolutional networks, which problem is further enlarged in 3D features. To compute this for 3D feature volume, the attentions between voxel and any other voxels need to be computed, i.e., Nx x Ny x Nz attentions for one voxel and Nx x Ny- x Nz x Nx x Ny x Nz attentions for all voxels, which is extremely large and hard to realized in regular GPUs.
4

---

Published as a conference paper at ICLR 2023
To deal with this problem and make the multi- head attention of 3D volumes feasible, we pro-
TSDF
pose to
sparse window structure to cal-
Down
Dilate Attn
Inv
Dilate Attn
culate the attention. As is displayed in Fig-
ure T in the medium and the fine level, sparsify the volumes using the occupancy pre-
3D 5-W-Attn
3D S-W-Attn
diction O², O and only compute the attention of the non-empty voxels. In addition, consider- ing that the nearby voxels contribute more to the shape of the current voxel and the distant voxels contribute less, we only calculate the at- tention within a local window of each voxel, as shown in Figure 2 Therefore, we are able to only calculate the multi-head attention of the occupied voxels within a small window, in which way the computation complexity is duced significantly.
Down
Dilate Attn
Inv
3D S-W-Attn
3D S-W-Attn
Down
Dilate Attn
Inv
3D S-W-Attn
3D S-W-Attn
Down
Dilate Attn
Inv
3D S-W-Attn
Global Attn
Specifically, for any non-empty voxel u; in the feature volume V, we first search all non-empty window centered on
Global Encode
voxels within n this voxel and get the neighbor voxels { E
Figure 3: The structure of the SDF transformer. "S-W- Attn" " denotes sparse window attention. (i)).. Then the query, key, and value embeddings are calculated as
Qi = L. (V(())),, K, = (V(e)))) = L.((()))
(4)
where Lup Lb, Ly are the linear projection layers.
For the position embedding P, we hope to block the influence from the scale of the 3D world coordinates. Hence we compute it based on the relative voxel position in the volume rather than based on the real-world coordinates (Mao et al. 2021). as
P3 C,(cg - v;).
(5)
Then the attention is calculated as Attention(u:)
SoftMax(Q:(K))+ P3) + P,).
(6)
JER(()
In this case, the computation complexity reduced from
Oap-Am = Nx x Ny x N2 x Nx Ny x N₂ x O(j),,
(7)
to
(8) where O(ij) is the complexity of one attention computation between voxel e; and Vy, Nowm is the number of occupied voxels in the volume, and Neccu is the number of occupied voxels within the local window. Assuming that the occupancy rate of the volume is 10% and the window size is 1 of the volume size. the computation complexity of the sparse window attention would be only
Osw Att = Noceu X Plece x O(ij);
1O
/1010
100000 of the dense 3D attention.
Ny
3.4 SDF 3D TRANSFORMER
Limited by the high resource- cosuurce-consmmnng of the multi-head attention, most of the previous works related to 3D transformers are only carefully performed on resource-saving feature processing, e.g., the one-off straightforward feature mapping without any downsampling upsampling (Wang et al.) 2021a), where the size of feature volumes remains unchanged, or the top-down tasks with only downsampling (Mao al., 2021), where the size of feature volumes is reduced gradually. In 3D reconstruction, however, a top-down- bottom-up structure more reasonable for feature extraction and prediction generation, in most of the 3D- DDD-CNN-assee structures (Murez al.) 2020 Sun et al., 2021 Stier et al.) 202T). So in this work, we design the first 3D transformer based hop-down-bottone up structure, is shown in Figure 3
hop-down-bottom-up
5

---

Published as a conference paper at ICLR 2023
Baseline
+ SDF Transformer
+ Post Dilate Attention
Ground Truth
Figure 4: Ablation study on the ScanNet dataset.
example, there are four feature levels in total, i.c. 3. 3. &1 16+ as shown in Figure In the encoder part, at each level, combination of downsampling and dilate-attention is proposed to downsample the feature volume. Then two blocks of the sparse window multi-head attention are used to aggregate the feature volumes. At the bottom level, global attention block is employed to make up the small receptive field of the window attention, and a global context encoding block is utilized to extract the global information. In the decoder part, we the inverse sparse 3D CNN to upsample the feature volume, i.e., we store the mapping of the down flow and now restore the spatial structure by inversing the sparse 3D CNN in the dilate-attention. Therefore, the final shape after the up flow should be the same as the input. Similar to FPN (Lin et al. 2017), the features in the down flow are also added to the upsampled features in the corresponding level. To enable the deformation ability, a pos--ilate-attentte is equipped after the down-up flow. Finally, a submanifold 3D CNN head with Tanh activation is appended to output the TSDF prediction. For the coarse volume V and medium volume V2, two and three-level of similar structures with Sigmoid activation are adopted.
Taking the network for the fine volume (V" in Figure(]
block
Dilate-attention. The direct downsampling of a sparse structure is prone to losing geometry struc- ture. To deal with this, between each level we first downsample the feature volume, and then dilate the volume with a sparse 3D CNN with the kernel size of 3. which calculates the output if any voxel within its kernel is non-empty. The dilation operation alone may also harm the geometry, since it may add some wrong voxels into the sparse structure. Thus we calculate the sparse window atten- tion of the dilated voxels, such that the voxels far from the surface would get low scores and do not contribute to the final shape. The dilated voxels are then joined to the downsampled volume by concatenating the voxels together. With this dilate-attention module, the 3D shape is prevented from collapsing. Without this module, the network performs badly and only generates a degraded shape.
Global attention and global context encoding. Since the attention blocks in the top-down flow are all londo-window based, there could be lack of the global receptive field. Considering the resolution of the bottom level is not high, we equip with global attention block at the bottom level, i... we calculate the attention between each non-empty voxel and any other non-empty voxel in the volume. This could build the long-range dependency missing in the sparse window attention blocks. In addition, we use the multi-scale global averaging pooling (Zhao et al., (017)) of scales 1, 3 to extract the global context code of the scene. This encoding module could aggregate the global information and explain the illumination, global texture, and global geometry style.
3.5 LOSS FUNCTION
The final TSDF prediction S is supervised by the log L1 distance between the prediction and the ground truth as L log S log S).
To supervise the occupancy predictions O², 0² in the coarse and medium levels, we generate the occupancy volumes based on the TSDF values. Specifically, the voxels with TSDF of -1 ~ 1 are regarded as occupied, and the values are set to 1, otherwise set to 0. Then a binary cross-entropy
loss is calculated between the prediction and the ground truth as: L = Od log O', 1=
To supervise the averaging weights W we use the occupancy in the back- paok-prooection following Stier et al. (2021). Intuitively, when the feature is back-projected from 2D image to the 3D space along the camera ray using multiple depth values, we hope the voxels close to the mesh surface have bigger weights in the fusion. Therefore. the 3D position is regarded between the project depth and the true depth from the depth map is smaller than the TSDF truncation distance. Then the cross entropy loss is applied the weights and the occupancy:
occupied if the difference
L! = 0 log or (W!), [= 1,2,3,
(9)
where a denotes Sigmoid, and 0 is the ground truth occupancy in the back-projection of image T

---

Published as a conference paper at ICLR 2023
Method
Acc L
Comp L
Chamfer
Prec f
Recall t
F-score t
VideMMMs Duzceker et al.
2021
0.079
0.133
0.106
0.521
0.454
0.474
Atlas [Murez el al
2020
0.068
0.098
0.083
0.640
0,539
0.583
NeuralRecon Sun et a
2021
0.054
0.128
0.091
0.684
0.479
0.562
VoRTX Stier et al.
2021)
0.054
0.090
0.072
0.708
0.588
0.641
Ours
0.049
0.068
0.058
0.754
0.664
0.705
Colmap Schönbbegger et al.
2016)
0.102
0.119
0.111
0.509
0.474
0.489
NeDepth Wang & Shen
2018)
0.129
0.083
0.106
0.443
0.487
0.460
GP-M GP-MVS (Hou et al.
2019a)
0.129
0.080
0.105
0.453
0.510
0.477
DPSNet (Im el a.
2019)
0.119
0.076
0.098
0.474
0.519
0.492
ESTDepth (ongg et al.
2021
0.127
0.075
0.101
0.456
0.5-22
0.491
DeepVideoMISS (Duzceker et al.
2021
0.107
0.069
0.088
0.541
0.592
0.563
Atlas MMurez et al.
2020)
0.072
0.076
0.074
0.675
0,605
0.636
NeuralRecon (Sun et al.
2021
0.051
0.091
0.071
0.630
0.612
0.619
TransformerFusion Bozic et al.
2021
0.055
0.083
0.069
0.728
0,600
0.655
Ours
0.032
0.062
0.047
0.829
0.694
0.754
Table I: Evaluation the 3D meshes on SeanNet. The upper part follows evaluation in Sun et al. while the lower part follows Bozic et al. ( The metric definitions are explained in the appendix.
2021
Method Abs Rel 4. Abs Diff 4 Sq Rel 4. RMSE + 6-1.25 f 6-1.25² 5-1.25²
Colmap (Scöönberger MVDepthNct Waan & Shen
2016
0.137
0.264
0.138
0.502
0.834
2018
0.098
0.191
0.061
0.293
0.896
0.977
0.994
GP-MVS (Hou et al.
2019a
0.130
0.239
0.339
0.472
0.906
0.967
0.980
DPSNet (Im et al.
2019
0.087
0.158
0.035
0.232
0.925
0.984
0.995
Atlas (Murez et al.
2020
0.065
0.124
0.043
0.251
0.936
0.971
0.986
NeuralRecon (Sun al.
2021
0.065
0.106
0.031
0.195
0.948
0.961
0.975
Vortx (Sher et al.
202T
0.061
0.096
0.038
0.205
0.9-3
0.973
0.987
Ours
0.051
0.086
0.033
0.199
0.958
0.980
0.990
Table 2: Evaluation of the 2D depth maps the ScanNet dataset. The upper part shows the results of depth- based methods, while the lower part shows volumetric methods, whose depths are rendered from the meshes.
4 EXPERIMENTS
4.1 EXPERIMENTS SETUP
Our work is implemented in Pytorch and trained on Nvidia V100 GPUs. The network optimized with the Adam optimizer (31 0.9, 82 0.999) with learning rate of 1 x 10-4, For a fair comparison with previous methods, the voxel size of the fine level is set to 4cm, and the TSDF truncation distance is
triple the voxel size. Thus the voxel size of the medium and the coarse levels are 8 cm and 16 cm, respectively. For the balance of efficiency and receptive field, the window size of the sparse window attention set to 10. For the view selection, we first follow Hou et al. (019b) to remove the redundant views, i.e., a new incoming frame is added to the system only if its relative translation is greater than 0.1 and the relative rotation angle is greater than 15 degree. Then if the number of the remaining views exceeds the upper limit, random selection is adopted for memory efficiency. The view limit set to 20 in the training, which means twenty images are input to the network for one iteration, while the limit for testing is set to 150. Our framework runs at an online speed of 75 FPS for the keyframes. Detailed efficiency experiments are reported in the supplemental materials.
ScanNet (Dai et al. 2017) a large-scale indoor dataset composed of 1613 RGB-D videos of 806 indoor scenes. We follow the official train/test split, where there are 1513 scans used for training and 100 scans used for testing. TUM-RGBI (Sturm et al.) 2012) and ICL-NUIM (Handa et al. 2014) are also two datasets composed of RGB-D videos but with small-number scenes. Therefore, following previous methods (Stier et al. 2021), we only perform the generalization evaluation of the model trained on ScanNet on these two datasets, where 13 scenes of TUM-RGBD and 8 scenes of ICL-NUIM are used.
4.2 EVALUATION
To compare with previous methods, we evaluate the proposed method on the ScanNet test set. The quantitative results are presented in Table/Tand the qualitative comparison are displayed in Figure
7

---

Published as a conference paper at ICLR 2023
NeuralRReco
Atlas
Ours
Ground truth
Figure 5: The qualitative results on the ScanNet datasct. Texture-less rendering is displayed in the appendix.
We first directly evaluate the reconstructed meshes with the ground-truth meshes, and obtain a sig- nificant improvement from previous methods, improving from F-score = 0.641 to F-score = 0. 705, shown in Table Then following [Bzzic al. (202T), add the same occlusion mask at evalua- tion to avoid penalizing more complete reconstruction, which is because the ground-trutt meshes are incomplete due unobserved and occluded regions, while our method could reconstruct more complete 3D shape, shown in Figure This results in a more reasonable evaluation, as in the sec- ond part of Table I The improvement is further enlarged, from F-score 0.655 to F-score = 0.754 compared to previous best method. The accuracy error is decreased from 0.055 0.032 m. which almost half (41.8%) of the previous best method, while the completeness error decreased by 25.3%, from 0.083 to 0.062 This owes to the feature aggregating ability of the proposed 3D SDF transformer, which can predict a more accurate 3D shape. This is also demonstrated in the generalization experiments ICL-NUIM and TUM-RGBD datasets, shown in Figure
After evaluating the reconstructed meshes, we also evaluate the depth accuracy of our method. Since method does not predict the depth maps explicitly, render the predicted 3D shape to the
8

---

Published as a conference paper at ICLR 2023
image planes and get the depth maps, following previous methods (Murez et al. 2020). The results are shown in Table 2 from which we can see our method decreases the error a lot from previous methods. The relative error is reduced by 16.4%, from 0.061 to 0.051. The accuracy of the depth maps also demonstrates the accurate feature analysis ability of the proposed 3D SDF transformer.
From the qualitative visualization in Fig- ure 5 we can see our method can predict complete and accurate 3D shape. Pre- vious methods which can recover a com- plete mesh usually reconstruct a smooth 3D shape with losing some details (Murez jet al., 2020). However, our method could predict more complete mesh than the ground truth, while the details of the 3D shapes are better recovered. Please note that for a fair comparison, the voxel size is set to cm, such that it is hard to recon- struct the geometry details less than 4 cm.
Method Acc |
Comp |
Prec t
Recall f F-score f
Atlas 0.175
0.314
0.280
0.191
0.229
NeuralRecon 0.215
1.031
0.214
0.036
0.058
VoRTX 0.102
0.146
0.449
0.375
0.408
Ours 0.083
0.142
0.522
0.390
0.447
Atlas 0.208
2.344
0.360
0.089
0.132
TUM
NeuralRecon 0.130
2.528
0.382
0.075
0.115
Vortx 0.175
0.314
0.280
0.194
0.229
Ours 0.129
0.455
0.406
0.173
0.254
Table Generalization experiments on the ICL-NUIM and TUM-RGBD datasets.
Method Acc L Comp 4 Prec t Recall f F-score t
4.3 ABLATION STUDY
Baseline 0.056
0.089
0.698
0.587
0.636
SDF transtormer. To verify the effec- tiveness of the proposed SDF transformer, first build baseline model with the same structure as Figure [ but the 3D SDF transformer is replaced by UNet structure of 3D CNN. Adding the variance fusion would improve the mesh in some clutter areas and slightly increase the per- formance. Then we add a base version of the SDF transformer, which does not include the global module and the post- dilate-attention module. The performance is significantly improved with this module, as is shown in Table $ and Figure 4 The reconstructed meshes possess much more geometry details compared to the baseline.
+ Var Fusion 0.054
0.090
0.713
0.594
0.647
SDF Former 0.036
0.065
0.807
0,671
0.732
+ Global 0.033
0.064
0.823
0.676
0.741
Post-Dila-AAtm 0.032
0.062
0.829
0.694
0.754
0.052
0.086
0.721
0.604
0.656
0.044
0.078
0.768
0.636
0.695
0.037
0.069
0.799
0.660
0.730
Window Size
8 0.033
0.065
0.822
0.682
0.746
10 0.032
0.062
0.829
0.694
0.754
Table 4: Ablation study on the ScanNet dataset. Components are added one by one in the upper part.
Global module. We next add the global module, including the bottom-leve global attention and the global context code. The sparse window attention block can only obtain the long-range dependency within a local window. Thus it may have problems when can not get enough information within this local window, e.g., the texture-free regions. Also, the global module could reason the global information like the illumination and the texture style.
Dilate attention. The dilate attention module is crucial in the SDF transformer, move all the dilate attention blocks. That will destroy the whole framework and generate a degraded 3D shape. Therefore, we only ablate the post dilate attention block after the down-up flow. This block could deform the shape and make it more complete, e.g., making up the crack as shown in Figure From the quantitative results in Table 4
not
we can also see the improvement of completeness.
we study the impact of the window size of the attention. It expected that larger window size would generate better result, since the range of the depen- dency is longer, but with the cost of more resource consumption. We choose 10 as the default size, considering that the performance improvement minor after that.
Window size. As shown in Table 4
5 CONCLUSION
We propose the first top-down-bottommuup 3D transformer for 3D scene reconstruction. A sparse win- dow attention module is proposed to reduce the computation, a dilate attention module is proposed to avoid geometry degeneration, and global module at the bottom level is employed to extract the global information. This structure could be used to aggregate any 3D feature volume, thus could be applied to more 3D tasks in the future, such 3D segmentation.

---

Published as a conference paper at ICLR 2023
REFERENCES
Aljaz Bozic, Pablo Palafox, Justus Thies, Angela Dai, and Matthias Nießner. Transformerfusion: Monocular rgb scene reconstruction using transformers. Advances in Neural Information Pro- cessing Systems, 10341444,,,, 2021.
Carlos Campos, Richard Elvira, Juan Gómez Rodríguez, José MM Montiel, and Juan D Tardós. Orb-slam3: An accurate open-source library for visual, visual-inertial, and multimap slam. IEEE Transactions on Robotics, 37(6):1874- 1890, 2021.
Rui Chen, Songfang Han, Jing Xu, and Hao Su. Point-based multi-view stereo network. In Pro- ceedings of the International Conference on Computer Vision, pp. 1538-1547, 2019.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations, 2020.
Arda Duzceker, Silvano Galliani, Christoph Vogel, Pablo Speciale, Mihai Dusmanu, and Marc Polle- feys. Deepvideomvs: Multi-view stereo on video with recurrent spatio natiootemppoal fusion. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 15324-15333, 2021.
A. Handa, T. Whelan, J.B. McDonald, and A.J. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In Proceedings of the IEEE International Conference on Robotics and Automation, Hong Kong, China, May 2014.
Yuxin Hou, Juho Kannala, and Amn Solin. Multi-view stereo by temporal nonparametric fusion. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 2651-2660, 2019a.
Proceedings of the International Conference on Computer Vision, pp. 2651-26600 2019b. Yuxin Hou, Juho Kannala, and Amo Solin. Multi-view stereo by temporal nonparametric fusion. In
Sunghoon Im, Hae-Gon Jeon, Stephen Lin, and In So Kweon. Dpsnet: End-to-end deep plane sweep stereo. In Proceedings of the International Conference on Learning Representations, 2019.
Mengqi Ji, Juergen Gall, Haitian Zheng, Yebin Liu, and Lu Fang. Surfacenet: An end-to-end 3d neural network for multiview stereopsis. In Proceedings of the International Conference on Com- puter Vision, pp. 2305-2315, 2017.
Michael Kazhdan and Hugues Hoppe. Screened poisson surface reconstruction. ACM Transactions on Graphics, 32(3):1 2013.
Patrick Labatut, J-P Pons, and Renaud Keriven. Robust and efficient surface reconstruction from range data. In Computer graphics forum, volume 28, pp. 2275 2290. Wiley Online Library, 2009.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. In Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 2117-2125, 2017.
Chao Liu, Jinwei Gu, Kihwan Kim, Srinivasa Narasimhan, and Jan Kautz. Neural rgb (r) sensing: Depth and uncertainty from video camera. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 10986-10995, 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hicrarchical vision transformer using shifted windows. In Proceedings of the International Conference Computer Vision, pp. 10012-10022, 2021.
10

---

Published as a conference paper at ICLR 2023
Xiaoxiao Long, Lingjie Liu, Wei Li, Christian Theobalt, and Wenping Wang. Multi-view depth estimation using epipolar spatio-temporal networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8258 8267, 2021.
William E Lorensen and Harvey E Cline. Marching cubes: A high resolution 3d surface construction algorithm. ACM siggraph computer graphics, 2144):163-169, 1987.
Jiageng Mao, Yujing Xue, Minzhe Niu, Haoyue Bai, Jiashi Feng, Xiaodan Liang, Hang Xu, and Chunjing Xu. Voxel transformer for 3d object detection. In Proceedings of the International Conference on Computer Vision, pp. 3164-3173, 2021.
Zak Murez, Tarrence van As, James Bartolozzi, Ayan Sinha, Vijay Badrinarayanan, and Andrew Rabinovich, Atlas: End-to-end 3d scene reconstruction from posed images. Proceedings of the European Conference on Computer Vision, pp. 414-431 Springer, 2020.
Richard A Newcombe, Shahram Izadi, Otmar Hilliges, David Molyneaux, David Kim, Andrew J Davison, Pushmeet Kohi, Jamie Shotton, Steve Hodges, and Andrew Fitzgibbon. Kinectfusion: Real-time dense surface mapping and tracking. In Proceedings of the International Symposium on Mixed and Augmented Reality, pp. 127-136. Icce, 2011.
Johannes L Schonberger and Jan-Michael Frahm. Structure- fronre--fom--oe from-motion revisited. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4104-4113 2016.
Johannes L Schönberger, Enliang Zheng, Jan-Michael Frahm, and Marc Pollefeys. Pixelwise view selection for unstructured multi-view stereo. In Proceedings of the European Conference on Computer Vision, pp. 501-518. Springer, 2016.
Noah Stier, Alexander Rich, Pradeep Sen, and Tobias Höllerer. Vortx: Volumetric 3d reconstruction with transformers for voxelwise view selection and fusion. In Proceedings of the International Conference on 3D Vision, pp. 320-330. IEEE, 2021.
Sturm, N. Engelhard, Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems, Oct. 2012.
Jiaming Sun, Yiming Xie, Linghao Chen, Xiaowei Zhou, and Hujun Bao. Neuralrecon: Real-time coherent 3d reconstruction from monocular video. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 15598-15607, 2021.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoe V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz. Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Informa- fion Processing Systems, 30, 2017.
Dan Wang, Xinrui Cui, Xun Chen, Zhengxia Zou, Tianyang Shi, Septimiu Salcudean, Z Jane Wang, and Rabab Ward. Multi-view 3d reconstruction with transformers. In Proceedings of the Interna- tional Conference on Computer Vision, pp. 5722-5731, 202la.
Kaixuan Wang and Shaojie Shen. Mydepthnet: Real-time multiview depth estimation neural net- work. Proceedings of the International Conference on 3D Vision, pp. 248 257. IEEE, 2018.
Peng Wang, Lingjie Liu, Yuan Liu, Christian Theobalt, Taku Komura, and Wenping Wang. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. Advances in Neural Information Processing Systems, 34:27171-27183, 2021b.
Haozhe Xic. Hongxun Yao, Shengping Zhang, Shangchen Zhou, and Wenxiu Sun. Pix2vox++: Multi-scale context-aware 3d object reconstruction from single and multiple images. Interna- tional Journal of Computer Vision, 128(12):2919229) 2020.
Yan Yan, Yuxing Mao, and Bo Li. Second: Sparsely embedded convolutional detection. Sensors, 18(1))33337, 2018.
11

---

Published as a conference paper at ICLR 2023
Yao Yao, Zixin Luo, Shiwei Li, Tian Fang, and Long Quan. Mvsnet: Depth inference for unstruc- tured multi- view stereo. Proceedings of the European Conference on Computer Vision, pp. 767-783, 2018.
Lior Yariv, Yoni Kasten, Dror Moran, Meirav Galun, Matan Atzmon, Basri Ronen, and Yaron Lip- man. Multiview neural surface reconstruction by disentangling geometry and appearance. Ad- vances Neural Information Processing Systems, 53:2492-25000 2020.
Lior Yariv, Jiatao Gu, Yoni Kasten, and Yaron Lipman. Volume rendering of neural implicit surfaces. Advances in Neural Information Processing Systems, 34:4805-4815, 2021.
Weihao Yuan, Yazhan Zhang, Bingkun Wu, Siyu Zhu, Ping Tan, Michael Yu Wang, and Qifeng Chen. Stereo matching by self-supervision of multiscopic vision. In Proceedings of the IEEF/RSJ International Conference Intelligent Robots and Systems, pp. 5702-5709. IEEE, 2021.
Weihao Yuan, Xiaodong Gu, Zuozhuo Dai, Siyu Zhu, and Ping Tan. Neural window cunly-connected crfs for monocular depth estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3916-3925, 2022.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 2881-2890, 2017.
12

---

Published as a conference paper at ICLR 2023
A APPENDIX
A.I MORE DETAILS
Volume sparsify. The ground-truth occupancy volumes are generated based on the ground-truth TSDF volumes. The voxels with TSDF value of [-1, 1] are regarded as occupied and set 1, otherwise set to 0. In the training or the inference, after the occupancy volume is predicted in the coarser level, the voxels of occupancy value less than 0.5 are regarded as empty and discarded, while the remaining voxels are regarded as occupied and transmitted to the next level. The sparse volume stored hash table, where the key of the table the hash value of the voxel, and the value of the table stores the corresponding feature. In the coarsest level, all voxels are regarded as non-empty and stored in the hash table, which does not consume much memory because the size of the volume small.
Training and inference. The training and inference are performed in a similar way. For a given sequence of images, first view selection is performed select images with translation greater than 0.1m and rotation greater than 15 degrees. Then a random selection is adopted from the remaining images if the number exceeds the upper limit. These images feature extraction, after which the features are fused to a 3D volume and fed to the 3D part to produce the TSDF volume prediction. The final mesh is extracted by marching cubes from the TSDF volume. This process the same as previous methods like Atlas, TransformerFusiio
then fed to the 2D backbone for
or VORTX.
For the number of the upper limit of the images, actually any number for the sequence length is okay for our framework, although more images lead to a better reconstruction of a scene. In our experiments, the number for training to 20 and the number for inference is set to 150.
Method
Per Frame Time
Per Scene Time
FPS
Atlas Murez
2020
ms
ms
NeuralRecon (Snn et al.
2021
30 ms
ms
33
TransformerPusion
Bozic ct al.
2021
130.5 ms
243.3 ms
7
VoRTX Sther
2021
71.4 ms
231.7 ms
14
13.3 ms
286.4 ms
75
Table 5: Efficiency experiments.
A.2 EFFICIENCY
The runtime analysis is presented in Table 5 For a fair comparison to previous methods, the time tested chunk of size 1.5 x 1.5 x 1.5 m with an Nvidia RTX 3090 GPU. Our framework consists of two parts: one the per-frame part, including the feature extraction of the 2D images; the other one is the per-scene part, including the feature fusion, 3D feature processing, and mesh generation. The per-frame model runs for every keyframe, i.e., keeps running whenever a new keyframe comes. Differently, the per-scene model runs only once for generating mesh reconstruction of scene, i.e., it only works after all frames are fed, or when we need to output a mesh. Therefore, the online speed of a normal running is 75 FPS, which only performs the mesh generation once the end.
A.3 METRICS
The definitions of the 2D metrics and 3D metrics used for evaluation are explained in Table
A.4 LIMITATIONS
Due to the volume representation, framework is limited by the trade-off between the resolution of the volume and the memory consumption. A smaller voxel size would cost much more memory. The voxel size is set 4 cm, such that the geometry details less than 4 cm hard to be recovered.
13

---

Published as a conference paper at ICLR 2023
Metries
Definition
Abs Rel
1 Σ ld d'1/d"
n
Abs Diff
1
Σ dd d=1
n
Sq Rel
1 E d dd12/dd
RMSE
# Σ dd - d-12
1.25'
Σ (max( $ ) < 1.25")
Acc
pe p (ming EP- ||p p (1)
Comp
m (minpe |pp p p'll)
Chamfer distance
Acc Comp
Prec
pe pe (ming eP ||p - p < 0.05)
Recall
meany* cP (minipe ||p p'|| < 0.05)
F-score
Recall
Piee
Table 6: Metric definitions. n denotes the number of pixels with both valid ground truth and prediction, d and d' denote the predicted and the ground-truth depths, p and p" denote the predicted and the ground-truth point clouds.
A.5 ROBUSTNESS TO THE POSE NOISE
Our method is based the given accurate camera poses, which is the same previous state-of- the-art methods like Atlas (Murez et al., 2020), NeuralRecon (Sun et al. (02T), and Transformer- Fusion (Bozic et al.] 2021), where the camera poses obtained by the standard SfM or SLAM systems. To inspect the robustness of our method to the pose errors, we add the Gaussian noise to the camera poses. A translation noise [Na,, Ny, N&] of N Gouss oT) is added to the trans- lation of the pose, while a rotation noise [Nrott, NNiithh, Nyaw] of N Gauss O0, R is added to the three angles of the pose. The metries following NeuralRecon (Sun al., 2021 are reported in Table 7 From the results. we can see our system can handle some translation errors but cannot handle the rotation errors well. But if the poses of only some frames are miscalculated, e.g., 10% of all frames, the performance decrease would be under control.
Ratio or (cm) OR (deg)
Acc Comp 1 Prec Recall
F-score
0
0
0
0.049
0.068
0.754
0.664
0.705
100%
0.5
0.050
0.068
0.745
0.658
0.698
100%
0.055
0.073
0.708
0.626
0.663
100%
0.5
0.084
0.117
0.525
0.446
0.480
100%
0
1
0.109
0.185
0.406
0.314
0.351
100%
0.5
0.5
0.084
0.117
0.516
0.435
0.471
100%
0.114
0.187
0.380
0.296
0.330
10%
0.5
0.5
0.055
0.074
0.715
0.629
0.668
10%
0.064
0.087
0.662
0.577
0.616
Table 7: Experiments with pose noise following NeuralRecon Sun et al.
2021 metrics.
A.6 RESULTS WITH SMALLER VOXEL SIZE
As expected, smaller voxel size leads to more accurate reconstruction but consumes much more GPU memory. We have trained the models with voxel sizes of 2cm and 3cm, but is hard to evaluate the models in the large scene of ScanNet test set, because the model of 2em requires too much memory of the GPU, Thus we only compare them on a medium scene, i.c., Scene-709, as reported in Table g The per-frame time remains unchanged while the per-scene time increases.
14

---

Published as a conference paper at ICLR 2023
Voxel Size Acc + Comp
Pree
Recall
F-score
Per Scene Time
4cm
0.033
0.053
0.837
0.730
0.780
286.4
3cm
0.025
0.061
0.882
0.756
0.814
435.6 ms
2cm
0.019
0.062
0.913
0.764
0.832
891.2 ms
Table 8: Evaluation of different voxel sizes on Scene-709
NeuralRecon
Atlas
Ours
Ground truth
Figure 6: Texture-less rendering the ground truth meshes for the qualitative comparison the ScanNet dataset.
A.7 MORE RESULTS
The texture-less rendering of the ground-truth meshes is shown in Figure 6 More results are pre- sented in Figure 7 and Figure
15

---

Published as a conference paper at ICLR 2023
NeuralRecon
Atias
Ours
Ground truth
Figure 7: More qualitative results.
16

---

Published as a conference paper at ICLR 2023
NeuralRecon
Atlas
Ours
Ground truth
Figure 8: More qualitative results.
17


Reviews:
[{"review_id": "s51ALeRaKN", "reviewer": "Reviewer_9DEs", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an SDF transformer network to improve monocular scene reconstruction. Firstly, the 3D transformer is introduced to aggregate the 3D features at different levels in a coarse-to-fine pattern. Secondly, a sparse window multi-head attention module is adopted to save computation costs. Thirdly, the dilate-attention structure, the global attention module, and the global context encoding module are designed to further improve the performance. The results of the ScanNet dataset demonstrate the effectiveness of the proposed method. ", "summary_of_the_review": "The paper aims to improve monocular scene reconstruction with 3D SDF transformers. A series of techniques are adopted to enhance the features. Experimental results validate the effectiveness of all the proposed modules. Considering the high quality of paper and incremental contributions, my current recommendation is marginally above acceptance.", "strength_and_weaknesses": "Strength: \n+ The approach is reasonable. \n+ Paper is well organized and easy to understand.\n+ Ablation study is extensive. And the results validate the proposed modules.\n\n\nWeaknesses:\n- The contributions are slightly incremental. Most approaches are introduced from existing works. The sparse window module is a standard technique in volume-based methods. Global attention and context encoding are commonly-used in 2D tasks. \n- The effect of dilation-attention is doubtful. Intuitively, the dilation and the downsampling may lose more geometry information, while the paper explains that the geometry structure benefits from dilation.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iAUQdJXI4f0", "reviewer": "Reviewer_Zo2i", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper aims for 3D reconstruction in a coarse-to-fine manner, and the key ideas include: 1) variance based 3D volume feature fusion; 2) sparse window multi-head attention; 3) dilate attention; and 4) SDF transformer backbone for the 3D volume.", "summary_of_the_review": "This work has good technique contributions, and the experiment is well done.", "strength_and_weaknesses": "Strengths:\n1. The approach is novel. Although 1), 2) and 4) are not ideas that are hard to figure out individually, it is good to put them together to build the 3D reconstruction system, and investigate their effects to the reconstruction quality. Moreover, I think idea 3) is something novel which might inspire or followed by other 3D sparse volume UNet style work if it is effective as described in the paper.\n2. The experiment is thorough. The comparison with other methods has a good coverage, and it shows a fair amount of improvement comparing to the previous arts. Also the ablation study clearly shows the benefits of each proposed component.\n\nWeaknesses:\n1. This work might not work very well with very small details due to the volume size (note that this is discussed in the appendix). However, it is actually not slow for offline reconstruction, and it would be interesting to see the results with smaller volume sizes and the analysis of the running time with regard to the volume sizes. Remember sparsity should help a lot for small volume sizes.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "h7HzfGv75o", "reviewer": "Reviewer_N9G6", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper describes a method to fuse coarse-to-fine Truncated Signed Distance Fields (TSDF) predictions over a number of time frames in an image sequence captured by monocular cameras. The TSDF are directly regressed from images with known poses at different scales, similarly to previous works such as NeuralRecon. The main contribution here is to introduce a transformer attention (e.g., instead of a GRU) module to fuse the coarse to fine TSDF volumes using sparse attention in the so-called \"top-down-bottom-up” approach. The experiments shows ablations and comparisons where the proposed approach performs well.", "summary_of_the_review": "My biggest concern is that the experiments and inference are not as well communicated. Despite this I am rating the paper above the acceptance threshold hoping that the authors will answer the questions/concerns.", "strength_and_weaknesses": "Strengths:\n\nThe problem statement is well explained and has a nicely written related work section.\n\nThe illustrations and explanations show efforts and consideration. The simplicity of the figures draw the reader’s attention to the key aspects of the paper.\n\nThe proposed architecture looks reasonable and ablation experiments have been provided for the empirical justification.\n\nThe qualitative results show where the proposed approach have highest gains clearly.\n\nWeaknesses:\n\nI have reservations on the way the paper has been posed as an independently explored approach for SDF fusion in relation to architectures such as NeuralRecon.\n\nDiscussions on what does not work when directly integrating the transformers instead of GRU would help the reader more. Some details in the experiments are missing\n\nSome of the writing contains rather “loosely written” phrases. Experiments writing can be improved. Minor problems exist in citations. E.g., Schonberger 2016 is not SLAM. NeuralRecon citation is missing in the experiments section.\n\nWhat is the sequence length used for the experiments at a time? Are the images input one by one or it requires a certain number, say 5 or 10 images for a single iteration of training. If so what is this length?\n\nHow is the inference performed? More details should be explicitly provided on inference.\n\nFigure 5 gt has texture which makes the comparison difficult. It is better to either have both or only the texture-less rendering for visualization.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AsZ-eWE0f2", "reviewer": "Reviewer_mQ2G", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The manuscript proposes to to use transformer layers applied to 3D voxel tokens at different levels of detail in order to improve the reconstruction quality of monocular 3D reconstruction (given frame poses) over related work that typically relies on 3D CNNs for multi-view aggregation and fusion.\n\nThe core contribution of the work is adapting the transformer paradigm to 3D reconstruction. This involves overcoming the cubic scaling of 3D space via sparse and windowed attention modules, and making sure the geometry does not degrade while downsampling via dilated attention. ", "summary_of_the_review": "\nMy main concern with the paper is the clarity of the description of the 3D Transformer network that I could not completely follow in full detail.  Otherwise the paper is is solid and makes a worthwhile contribution to the community.  If the clarity problems can be resolved, I think this is a good paper that should be accepted. Until then I am leaning marginally towards reject.", "strength_and_weaknesses": "The paper is overall well written with some problems in the model section. The illustrations are well done and help communicate the system and properties of the components. Figure 2 was important for me to understand the dilated attention (dilated can have different meanings as well!) On that note I would consider changing the name to something else. Dilation in CNNs is usually a step size between pixels/features used at the input into the kernel. But here we are growing the set of 3d occupied 3d voxels. Its a bit different.\n\nOne of the main weak points to me is the explanation of the SDF-Transformer model. \n- In Fig3 there are some arrows from the left to the right side of the multi-scale transformer system at the level of the 3D S-W-Attn blocks. As far as I can tell there is no explanation of what that connection does? Does it mean we are using the feature volume on the down-sample path as the queries in the upsample path? \n- Also the connections from the feature volumes after Fusion (V^i in Fig 1) are not shown in Fig3. I assume they are also arrows into the 3D S-W-Attn module on the downsample-side (left) of the diagram? \n- Also I am assuming 3D S-W-Attn stands for Sparse Window attention module? Please clarify in the text. \n\nI am also not 100% sure about the dilated attention. Specifically I am confused about this sentence:\n> Then we calculate the sparse window attention of the dilated voxels and join them into the downsampled volume\n\nHow is the volume downsampled? Local averaging?  What does it mean to join them into the downsampled volume? Does it mean we are running some kind of attention mechanism? If so what are keys and values for this one? Or is the attention map inside the self attention module modified?  Or does this just mean that for the purposes of the S-W-Attn module we just append the list of dilated voxels to the list of occupied voxels?  If we dilate the attention map to extend over non-occupied voxels how do we have features in those voxels? Or is this not using a sparse feature volume? \n\nIn the feature volume construction Eq 1 is called variance but it looks more like a squared deviation from the mean? Usually the sample variance would be computed as 1/(N-1) \\sum_i (V_i - V_mean)^2 ? So more like the \"total variance\" in Eq 3. Just wanting to double check on the notations here. \n\n\n The quantitative evaluation is well done. Comparison with major related work on the key datasets.  The ablation studies are important and clearly support the use of the 3D Transformers over 3D CNNs. Since this is the core contribution these ablations are key. \n\nLots of qualitative visualizations illustrate what to expect from the approach.  However, in the qualitative comparisons the GT mesh should be shown without color so we can better compare the geometric accuracy.  While I do see that the gains from larger window size seem to saturate from 8->10 I would still have liked to see some higher window sizes as well.  There is an opportunity to show the run-time tradeoff in this ablation. I.e. how much slower does the model get when we go from 8->10 or 10->12 window size? I think that would make it pretty clear why we want to stick with 10?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster