TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
Authors: Lukas Koestler
Nan Yang
Niclas Zeller
Daniel Cremers
Venue: CoRL2021 Poster (2021)
Keywords: SLAM
Dense Mapping
Multi-view Stereo
Deep Learning

Abstract: In this paper, we present TANDEM a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photometric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose a novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that is built incrementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet (CVA-MVSNet) that utilizes the entire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Finally, the predicted depth maps are fused into a consistent global map represented as a truncated signed distance function (TSDF) voxel grid. Our experimental results show that TANDEM outperforms other state-of-the-art traditional and learning-based monocular visual odometry (VO) methods in terms of camera tracking. Moreover, TANDEM shows state-of-the-art real-time 3D reconstruction performance. Webpage: https://go.vision.in.tum.de/tandem

TANDEM: Tracking and Dense Mapping in Real-time using Deep Multi-view Stereo
Lukas Koestler!
Nan Yang!
Niclas Zeller
Daniel Cremers 1.2
equal contribution
corresponding author
Technical University of Munich 2 Artisense
Karlsruhe University of Applied Sciences
Abstract: In this paper, we present TANDEM, a real-time monocular tracking and dense mapping framework. For pose estimation, TANDEM performs photo- metric bundle adjustment based on a sliding window of keyframes. To increase the robustness, we propose novel tracking front-end that performs dense direct image alignment using depth maps rendered from a global model that built in- crementally from dense depth predictions. To predict the dense depth maps, we propose Cascade View-Aggregation MVSNet tire active keyframe window by hierarchically constructing 3D cost volumes with adaptive view aggregation to balance the different stereo baselines between the keyframes. Finally, the predicted depth maps are fused into a consistent global map represented as truncated signed distance function (TSDF) voxel grid. Our experimental results show that TANDEM outperforms other sfate-of-the-art tradi- tional and learning-bast based monocular visual odometry (VO) methods in terms of camera tracking. Moreover, TANDEM shows stateof-the-aan real-tims 3D recon- struction performance. Webpage: attps://gg vision. in. tum. de/ /tandm
that utilizes the en-
Keywords: SLAM, Dense Mapping, Multi-view Stereo, Deep Learning
Introduction
Real-time dense 3D mapping is one of the major challenges in computer vision and robotics. This problem, known as dense SLAM, includes both estimating the 6DoF pose of sensor and dense reconstruction of the surroundings. While there exist numerous well-working and robust RGB-D signifi-
mapping solutions [1, 2, 3], real-time dense reconstruction from monocular cameras cantly more difficult challenge as depth values cannot be simply read out from the sensor and fused. Nevertheless, a very important problem, as monocular approaches offer significant advantages over RGB-D-based methods [1] which are usually limited to indoor environment due to the near- range sensing or LiDAR-based [4] solutions which are expensive and heavyweight.
Several deep neural network (DNN) based approaches have been proposed to tackle the problem of monocular tracking and dense mapping by utilizing monocular depth estimation [5], variational auto-encoders [6. 7, 8), or end-to-end neural networks [9, 10). Unlike the aforementioned works, in this paper, we propose a novel monocular dense SLAM method, TANDEM, which, for the first time, integrates learning- based multi-view stereo (MVS) into traditional optimization- based VO. This novel design of dense SLAM shows state-of-the-ar tracking and dense reconstruction accuracy as well as strong generalization capability on challenging real-world datasets with the model trained only on synthetic data. Figure shows the 3D reconstructions delivered by TANDEM on unseen sequences.
Our contributions. (1) novel real-time monocular dense SLAM framework that seamlessly cou- ples classical direct VO and learning-based MVS reconstruction; (2) to our knowledge, the first monocular dense tracking front-end that utilizes depth rendered from a global TSDF model: (3) novel MVS network, CVA-MVSNe. which is able to leverage the entire keyframe window by utilizing view aggregation and multi stage depth prediction; (4) of-te-of-th-ase art tracking and recon- struction results on both synthetic and real-world data.
5th Conference on Robot Learning (CoRI. 2021), London, UK.

---

Figure 1: TANDEM is a monocular dense SLAM method that estimates the camera poses and reconstructs the 3D environment in real-time. The figure shows the estimated camera trajectories and the dense 3D models on the sequences our Replica [11] test split (UL., UR), EuRoC (12) (BL), and ICL-NUIM [13] (BR) using model trained on the synthetic Replica dataset.
Related Work
There are two different work streams related to the proposed method. On one side, there is pure 3D reconstruction based on posed images and, on the other side, there are full SLAM or VO frameworks that simultaneously estimate camera poses and a 3D reconstruction of the environment.
3D Reconstruction. Most dense 3D reconstruction approaches consider images and corresponding reference poses as inputs and estimate a dense or partially dense reconstruction of the environment. Over the last decade, several classical methods have been proposed [14, 15, 16, 17].
Recently, deep-learning-basee
methods have shown superior performance over classical methods. These methods regress a 3D model of the environment utilizing DNNs. This 3D model can be either in the form of volumetric representation [9, 18, 19, 20], a 3D point cloud [21] or a set of depth maps [22, 23, 24]. Nowadays, most popular are methods which predict the final model from 3D cost volumes. Huang et al. [22] proposed one of the first seaa-volume-bssee
approaches. Zhou et al. [24] aggregate multiple image-pair-wise volumes to single cost volume and use a 3D CNN for depth prediction. Yao et [23] propose to directly calculate a single volume based on 2D deep feature maps predicted from the input images. In follow-up work, Yao al. [25] replace the depth pre- diction CNN by a recurrent network. To improve run run-time and memory consumption, Gu et al. [26] propose cascade cost volume. Yi et al. [27] introduce self-adaptive view aggregation to weigh the voxel-wise contribution for each input image. The proposed CVA-MVSNet is built upon the two aforementioned works [26, 27] and largely inspired by them. However, only by their combination and adaption to the SLAM setting we achieve better performance and real-time capability.
While all previous methods are based on set of selected frames, Murez et al. [9] instead directly predict a TSDF model from a single global 3D cost volume. Weder et al. [18] propose a learning- based alternative to classical TSDF fusion of depth maps. While these volumetric representations, in general, are rather memory intense, Niessner et al. [28] propose voxel- sxel-hhahingg to overcome this limitation and Steinbrücker et al. [29] perform depth map fusion CPU using octree.
RGB-D SLAM. In the area of visual SLAM, RGB-D approaches by nature provide dense depth maps along with the camera trajectory and therefore target to solve similar problem as our ap- proach. Bylow al. [30] and Kerl al. [2, 31] mainly focus on accurate trajectory estimation from RGB-D images. In addition to camera tracking, Newcombe et al. [1] integrate the depth maps into a global TSDF representation. Whelan et al. [3] perform surfel-basee based fusion and non-rigid surface deformation for globally consistent reconstruction. Kähler et al. [32] use a TSDF map representation which split into sub maps model loop closures. While most previous methods optimize only for the frame pose, Schöps et al. [33] propose a full bundle adjustment based direct RGB-D SLAM

---

Roptioma Wer
Exy
Jup.
Opinaadd Polts
Avee
on
Traakig
Dec
Warsig
(a) TANDEM System Overview.
(b) CVA-MVSNet Architecture.
Figure 2: (a) Every frame is tracked using the optimized sparse points from the visual odometry and the dense depth map rendered from the 3D model. The poses of the keyframes are estimated by sliding-window photometric bundle adjustment and fed into the CVAeMV for dense depth pre- diction. The depth maps are fused into globally consistent TSDF volume. (b) CVA-MVSNel builds cascaded cost volumes and hierarchically estimates the depth maps. The view aggregation module effectively aggregates the features of the multi-view images by predicting self self-adaptive weights.
which optimizes for both camera pose and 3D structure. Sucar et al. [34] integrate DNN-based implicit scene representation into an RGB-D SLAM system.
Monocular SLAM. Compared to RGB-D methods, for monocular approaches both tracking and mapping become much more challenging. Using a single monocular camera, Newcombe et al. [35] perform optimization based on photometric cost volume to jointly estimate the camera pose and dense depth maps real-time on a GPU. Engel et [36] propose the first large-scale photometric SLAM formulation including loop closure detection and pose graph optimization. By using sparse representation, Engel et al. [37] were able to formulate the first fully photometric VO framework which jointly estimates pose and depth in real-time. To obtain denser reconstructions, Mur-Artal et al. [38] perform semi-dense probabilistic depth estimation on top of feature-based SLAM [39]. Schöps al. [40] perform temporal, plane-sweep-banee based depth estimation using the poses and images obtained from a mobile tracking device. Tateno et al. [5] and Yang et al. [41, 42] leverage DNNs in a traditional direct SLAM framework improve tracking performance and overcome the problem of scale ambiguity. While traditional geometric and hybrid approaches still achieve superior tracking performance, there several fully learned SLAM frameworks [43, 44, 45, 24], which are superior in terms of reconstruction completeness. Jatavallabhula et al. [46] propose a differential optimizer which has the potential to bridge the gap between traditional and learning-based SLAMM. A novel idea for deep-lcarning -ased SLAM is proposed by Bloesch et al. [6]. The authors propose to learn frame-wise code representation for the scene depth, which is the jointly optimized together with the camera pose. The work of Czarnowski et al. [7] is an extension of [6], where the code representation integrated into full SLAM system. Zuo al. [8] make use of similar code representation in visual-inertial setup and furthermore feed sparse, tracked features into a DNN.
TANDEM
The proposed TANDEM is comprised of three components: monocular visual odometry (sec- tion 3.1), dense depth estimation with CVA-MVSNet (section 3.2), and volumetric mapping (suppl.). Figure 2a shows an overview of the system. The visual odometry utilizes the monocular video stream and the dense depth rendered from the 3D TSDF model to estimate camera poses in a sliding-windus manner. Given the keyframes and their estimated poses, the proposed CVA-MVSNe predicts dense depth map for the reference keyframe. To reconstruct complete and globally consistent 3D model of the environment, the depth maps are then fused into the TSDF voxel grid [47] with voxel- hashing [28]. By seamlessly integrating these components, the resultant system TANDEM enables real-time tracking and high-quality dense mapping from a monocular camera. Further details, in- cluding the TSDF volume initialization, are given in the supplementary material.
3.1 Visual Odometry
Estimating camera poses by tracking a sparse set of points across multiple frames has shown great performance in recent. VO systems [37, 39]. Using more points for the joint optimization, how- ever, does not necessarily further improve the accuracy of the estimated poses while significantly

---

increases the runtime [37]. Therefore, in the proposed VO system we make use of a direct sparse windowed optimization back-end as described in Direct Sparse Odometry (DSO) [37]. However, we utilize dense depth maps rendered from the global TSDF model, which we build up incrementally, in the direct image alignment front-end. In numerous experiments, we confirm that this combination of dense tracking front-end and sparse back-end optimization significantly improves the tracking performance (cf. Table 1) while maintaining a fast runtime.
Dense Front-end Tracking. The front-end tracking provides camera-rate pose estimations and serves as initialization for the windowed optimization back-end. In the original DSO, a new frame is tracked against the last keyframe by direct image alignment using sparse depth map DDS0 [p] generated from all points in the optimization window. This approach, however, lacks robustness (cf. Table 1) due to the sparsity of the depth map. We alleviate the issue by incorporating dense depth map DTSDF which rendered from the constructed TSDF model. For each pixel p in the current keyframe n, we assign a depth value either based on the sparse VO points DDSO [p). if available, or based on the rendered dense depth D [p), otherwise. Due to the increminntall--uen --biill TSDF model, the combined depth buffer might not contain valid depth values for all pixels but it is much denser in comparison to using the sparse depth values only. The nearly-dense combined depth map is used for two-frame direct image alignment.
3.2 CVA-MVSNet
Let [T, T.)) be the set of active keyframes where I, is the image of size (H, W) and Tr is the corresponding global pose estimated by the VO. CVA-MVSNet is based the principles of multi-view stereo [48] and further leverages deep neural networks [23] to estimate a depth map for the reference frame In CVA-MVSNet overcomes the prohibitive memory requirement of deep MVS networks by hierarchically estimating the depth using cascaded cost volumes and aggregates the deep features of all the keyframes effectively with a self-adaptive view aggregation module.
As shown in Figure 2b, the multi-scale deep features F; of the keyframes are firstly extracted by 2D U-Nets with shared weights, where i E [1, n) is the frame index and s E [1.3] is the scale index. As a result, F; is of the shape (F", H=, W*) where 1 is the feature dimension of the scale s, H = H/2 2T/23-2, and W W/23-5 The depth map of the reference frame is estimated hierarchically with 3 stages each of which takes the set of features [F:} I as the inputs and predicts the reference depth map D' of shape (H*, W*). For clarity, we first explain how single stage estimates the depth and then describe how multiple stages are assembled hierarchically.
Single Stage Depth Estimation. For each stage, cost volume Ca needs to be constructed using the deep features - For each pixel of the reference frame, we define D depth hypothe- ses, which results in a tensor Diup of shape (D*, H", W*). The deep features F; of each frame are geometrically transformed with differentiable warping [49] using the depth hypotheses, the rel- ative pose T = T-IT; and the camera intrinsics. As a result, a feature volume V of shape (F*,D*, D", W*) is constructed for every frame.
In order to aggregate the information from multi-view feature volumes into one cost volume C*, most prior deep MVS methods treat different views equally and use a variance-based cost metric:
C Ei (V: Voja
E..V V
where
However, in the sliding-window SLAM setting, the keyframes are not evenly distributed within the optimization window - typically the distance between newer keyframes is much smaller than between older keyframes. This causes considerable occlusion and non-overlapping images. The variance-based cost volume, which weighs different views equally, is thus inappropriate. To alleviate this issue, we employ self-adaptive view aggregation [27] to construct the cost volume:
C = E. (1 Wi) O (V. VV2²
(2)
where the view aggregation weights W have the shape (1, D=, (W') and @ is element-wise multiplication with broadcasting. The view aggregation weights W are estimated by shallow 3D convolutional network for each Vi separately by taking (V: V3)² as the input. This aggregation module allows the network to adaptively downweight erroncous information.

---

The cost volume C is then regularized using 3D U-Net and finally passed through a softmax non-linearity to obtain a probability volume P* of shape (D*, H W Given the per-pixel depth hypotheses D hyp of shape (D*, H*, W*) the estimated depth is given as the expected value
D
1* w|
Σ d=1
P' h, wi Diypld, Id, h, w].
(3)
Hierarchical Depth Estimation. The network leverages the depth estimated from the previous stage (x > 1) to define a fine-grained depth hypothesis tensor D hyp with a small Dr. Since no prior stage exists for the first stage, each pixel of Digr has the same depth range defined by [dmén: dmaz] with D 48 depth values. For the later stages (s 1). the depth D* upsampled and then used as prior to define Diy Specifically, for the pixel location (h, w). Diup (,, h, w) is defined using the upsampled D (h, w) as the center and then sampled D values around it using a pre-defined offset [26]. In this way, fewer depth planes are needed for the stage with a higher resolution, i.e., D 2 D D We train the network using the L.1 loss applied on all three stages with respect to the ground-truth depth and use the sum as the final loss function.
3.3 Implementation Details
To guarantee real-time execution, TANDEM leverages parallelism on multiple levels. Dense track- ing and bundle adjustment are executed in parallel threads on the CPU while, asynchronously and in parallel, TSDF fusion and DNN inference are run on the GPU. We train our CVA-MVSNet in PyTorch [50] and perform inference in C++ using TorchScript.
TANDEM can processes images at ca. 20 FPS while running on a desktop with an Nvidia RTX 2080 super with 8 GB VRAM, and an Intel i7-9700K CPU. This includes tracking and dense TSDF mapping but no visualization or mesh generation. We refer to the supplementary for further details, including how to potentially scale the network for deployment on embedded platforms.
4 Experimental Results
As TANDEM is complete dense SLAM system, we evaluate it for both monocular camera tracking and dense 3D reconstruction. Specifically, for the camera tracking, compare with state-of-the- art traditional sparse monocular odometry, DSO [37] and ORB-SLAM [39], as well as saaming-bbsse dense SLAM methods, DeepFactors [7] and CodeVIO [8]. For the 3D reconstruction,
compare with a state-of of--the-rt deep multi-view stereo method, Cas-MVSNet [26], end-to-end reconstruction method, Atlas [9]. learning- based dense SLAM methods, CNN-SLAM [5]. DeepFactors [7], and VIO [8], as well as iMAP [34], recently proposed RGB-D dense SLAM method using a deep implicit map representation [51, 52].
In the following, we will first introduce the datasets we use for training and evaluation. Note that only CVA-MVSNet needs to be trained while the dense tracking part of TANDEM is purely optimization- based and does not require training on specific datasets. Then, the ablation study for TANDEM demonstrates different design choices. In the end, we show quantitative and qualitative comparisons with other trate-of-hhe-ae methods. Due to the limited space, we demonstrate only part of the conducted evaluation and refer to the supplementary material for additional experiments.
4.1 Datasets
Training sets. We train two models for CVA-MVSNet: One on the real-world ScanNet [53] dataset and the other one on the synthetic Replica dataset [11]. The ScanNet dataset consists of 1513 indoor scenes and we use the official train and test split for training CVA-MVSNet give a fair comparisons with DeepFactors and Atlas. However, the geometry and texture in ScanNet show noticeable artifacts and incompleteness [11], which limits potential to training high-quality dense reconstruction methods. To complement the ScanNet dataset, we build upon the recently proposed Replica dataset [11], which consists of 18 photorealistic scenes. These scenes were captured from real-world rooms using state-of-thee of-th-a-t 3D reconstruction pipeline and show very high-quality texture and geometry. Because the authors did not release any sequences, we extend the dataset by manually creating realistic camera trajectories that yield 55 thousand poses and images.
5

---

Image
DeepFactors
Atlas
TANDEM (Replica)
GT Depth
iclloffccoo
Depth [m)
euroc idallingo
Depth [m]
Dopth [m]
Figure 3: Depth comparison for DeepFactors [7], Atlas [9], and TANDEM unseen sequences. TANDEM produces finer-scale details, c.g. the plant in the second row, or the ladder in the third row. For EuRoC only sparse ground-truth depth available. high-resolution version this figure can be found in the supplementary material.
Pulled VS0
ICE-NIUM Ivtng?
Allao (Qousetty)
TANDEN (Gonerty)
TANDEM [Tesses])
Allas (Coonetey)
"LANDEM (Geautty)
TANDEN [Teakel]]
Figure 4: Qualitative comparison of Atlas [9] and TANDEM on unseen sequences. Atlas does not construct textured meshes, so we also render the pure geometry from TANDEM for comparison.
Evaluation sets. We use the ICL-NUIM [13] dataset and the Vicon Room sequences of the EuRoC dataset [12] for evaluating the tracking and the dense 3D reconstruction. Note that TANDEM is not trained either of the datasets. ICL-NUIM is synthetic indoor dataset with pose and dense depth ground-truth. contains low-texture scenes which are challenging for monocular visual odome- try and dense depth estimation. EuRoC is real-worke dataset recorded by miero aerial vehicle (MAV).
4.2 Camera Pose Estimation
We evaluate TANDEM for pose estimation against other state-of-the-arr monocular SLAM methods on EuRoC and ICL-NUIM. On EuRoC, we evaluate against DSO [37], ORB-SLAM2 [39], Deep- Factors [7], and CodeVIO [8] uses camera and IMU sensor. Note that we off the global optimization and relocalization of ORB-SLAM2 for a fair comparison. We also implement a vari- of DSO (DSO Dense Depth) which uses all the pixels of the dense depth maps estimated by CVA-M CVA-MVSNet for the front-end direct image alignment. Note that the difference between TANDEM and DSO Dense Depth that TANDEM tracks against the global 3D model by rendering the depth maps from the TSDF grid. TANDEM achieves overall better tracking accuracy and robustness than the other monocular methods on both ICL-NUIM and EurRoC. Due to the limited space, we show the results on EuRoC in Table of the main paper and kindly refer to the supplementary material for the results on ICL-NUIM where we also show the comparision with DeepTA.M [24].
All the methods except for CodeVIO [8] are run five times for each sequence and reported with the mean RMSE error and standard deviation in terms of absolute pose estimations after Sim(3) align- ment with ground-truth For Code we directly take the numbers reported in their paper. The comparison with DSO and DSO Dense Depth indicates that the proposed dense tracking against the global 3D model improves the camera pose estimations, especially on the more challenging se- quences (V102 and V202). However, should admit that TANDEM still cannot compete with CodeVIO which uses an IMU sensor for pose estimations.
4.3 Ablation Study
We conduct the ablation study of CVA-MVSNet on the test split Replica and show the re- sults in Table 2. Specifically, we evaluate the effectiveness of using the full VO window with

---

Table 1: Pose evaluation on EuRoC [12]. All the methods are Sim(3) aligned w.r.t. the ground-truth trajectories. The mean absolute pose errors and the standard deviations over five runs are shown.
Sequence
CodeVIO18]
DeepFactors[7]
DS0137)
ORB-SLAM2 (39)
DSO+Dense Depth
Ours
10RoC/V101
0.06 (-)
1.48 (34.134)
(200000)
0.31 (#0022
0.09 (=00.002
0.09 (==0.001)
BuRcC/V102 102
0.07 (-)
Lost
0.27 (+0.017)
0.11 (#0005
0.28 (=0.015)
0.17 (#:0.006)
BaRoC/V201
0.10
1,06 (=0.44)
0.09 (2:0.005)
1.40 (1:0.211)
0.09 (2:0.005)
0.09 (1:0.002)
BuRoC/V202
0,06 (-)
1,89 (=0.019)
0.21 (#000200
0.84 (#0.648)
0.19 (#0.022)
0.12 (+0.009)
Table 2: Ablation study of CVA-MVSNet on Replica [11]. Using all keyframes within the VO window (Win) does not improve the baseline. However, combining Win with view aggregation (VA) yields more accurate results at the cost of increased inference runtime and memory. By reducing the number of depth planes (S) from (48, 32, 8) to (48, 4, 4) we retain high quality and guarantee the real-time performance of TANDEM. Best shown in bold and second best shown underlined.
Win
VA
S
Absl. [cm]
a1 f [%]
02 [%]
dg 1 [%]
Time.
Mem [MiB]
[ms]
baseline [26]
2.64
98.64
82.33
20.12
142
3447
+ VO window
2.64
98.39
83.55
20.56
215
4117
+ View aggregation
4
1.92
99.00
88.59
26.03
288
4117
+ Fewer depth planes
2.33
98.51
86.93
26.09
158
2917
keyframes (Win), the view aggregation module (VA), and fewer depth planes (S) with (48, 4, 4) for (D', D2, Da). The baseline method is the original Cas-M using 3 multi-view images as the inputs, no view aggregation module, and more depth planes (48, 32, 8). We use the absolute differ- ence (Abs), and the percentage of inliers with different thresholds (a), 02. ag) the metrics for the depth map evaluation. Please refer to the supplementary material for the formulas of the metrics. In addition, also show the inference time and the memory usage of different models. From the table, we can see that using the entire keyframe window with more frames does not improve the accuracy over the baseline model while increasing the runtime and memory usage. With the view aggregation module, the accuracy is significantly improved, but the runtime further increases. Using fewer depth planes does not show a significant drop in accuracy but improves the runtime and memory usage a lot. Therefore. to guarantee the real-time performance of TANDEM, we use the fewer-plane model as the final CVA-MVSNet and all other experiments in the paper are conducted with this model.
4.4 3D Reconstruction
We evaluate the reconstruction accuracy on both ICL-NUIM and EuRoC. On ICL-NUIM we com- pare with DeepFactors [7], CNN-SLAM [5], Atlas [9], and Cas-MVSNet [26]. Table shows the evaluation results. Since Atlas, pure 3D reconstruction method, does not estimate poses, we pro- vide ground-truth poses as the input. Note that Atlas estimates a TSDF volume directly from a 3D CNN, so render the depth maps for evaluation against other methods. CNN-SLAM, Deep- Factors, and Atlas are trained on ScanNet. For Cas-MVSNet, we re-train it on the Replica dataset and use the same poses for our CVA-MVSNet. We show the results of the evaluations for our ScanNet-trained model and our Replica-trained model to facilitate a fair comparison. The depth maps of monocular methods aligned in scale based on the trajectory and further details are given in the supplementary. We use the a metric as the major measurement for accuracy. From Table 3, can see that method shows notable improvement in comparison the other methods and delivers the best result on average. Note that CVA-MVSNet achieves better results with the model trained on the synthetic Replica than the model trained on ScanNet.
On EuRoC, we cannot compare with CNN-SLAM since the numbers on EuRoC are not provided on the paper and code is not publicly available. We further add CIO [8] into the evaluation on EuRoC as it is a recent dense SLAM system and it was also evaluated on EuRoC. Please note that Code uses monocular camera and an inertial sensor for tracking, while TANDEM and other SLAM methods rely only on monocular cameras. Table 4 shows the evaluation results.
We further evaluate TANDEM against iMAP [34], an RGB-D dense SLAM system that leverages deep implicit map representation. Please note two major differences between iMAP and TANDEM: on one hand, iMAP uses an RGB-D sensor while TANDEM needs only monocular camera; on
7

---

Table 3: Depth evaluation on ICL-NUIM [13]. We show the percentage of pixels for which the estimated depth falls within 10% of the groundtruth value.
CNA-SLAM
DeepFactors [7]
Atlas
Cas-MV
Ours (ScanNet)
Ours (Replica)
Sequence
[5]
[9]
[26]
icl/officeO
19.41
30.17
28.79
77.73
52.34
84.04
icl/officel
29.15
20.16
62.89
88.87
61.83
91.18
icl/livingo
12.84
20.44
83.16
97.06
86.42
97.00
icl/living
13.04
20.86
36.93
84.11
71.35
90.62
Average
19.77
30.17
66.93
86.94
67.99
90.71
Table 4: Depth evaluation on EuRoC [12]. We show the percentage of correct pixels d1 as in [54].
Ours (ScanNet)
Ours (Replica)
Sequence
CodeVIO[8]
DeepFactors|7]
Atlas[9]
Cus-MVSNer(26]
EuRoC/V100 101
86,99
71.82
57.16
93.05
93.69
94,25
EuRoC/V 102
78.65
92.29
88,03
89.62
90.50
EuRoC/V201
77.32
71.85
93.28
95.43
96,84
97.17
EuRoC/V202
71.98
68.26
64.33
93.16
92.65
95.68
Average
78.74
76.77
92.42
93.20
94.40
Table 5: Comparison to iMAP [34]. TANDEM shows comparable performance to iMAP, which uses RGB-D data, but no training prior to scanning. The mesh-basee 3D metries are as in [34].
room-1
office-4
Acc [em]
Comp [cm]
[%]
Acc [em]
Comp [em]
CR [%]
iMAP TANDEM
3.69
4.87
83.45
4.83
6.59
77.63
4.26
4.71
81.95
3.76
6.11
74.66
the other hand, the DNN of iMAP does not require any pre-training and it is trained purely online with the RGB-D inputs, while the depth estimation network TANDEM requires offline training. Since iMAP was also evaluated Replica, we, therefore, compare TANDEM with iMAP the two sequences of their dataset which are not included in the training set of our Replica split. We use the evaluation metrics from iMAP and show the results in Table 5. In general, TANDEM achieves similar results to iMAP while using a monocular camera.
In Figure 3 we show qualitative depth maps estimated by DeepFactors, Atlas, and TANDEM. Both DeepFactors and Atlas can recover the geometry of the underlying scene well, but our method gener- ally manages to capture more fine-scale details. We further show the complete scene reconstruction as meshes in Figure 4. As DeepFactors does not generate a complete 3D model by itself, we only compare TANDEM with Atlas for this experiment. From the figure we see that, similarly to the depth maps, TANDEM is able to reconstruct more fine-scale details than Atlas.
Conclusion
We presented TANDEM, real-time dense monocular SLAM system with novel design that cou- ples direct photometric visual odometry and deep multi-view stereo. In particular, we propose CVA-MVSNet which leverages the whole keyframe window effectively and predicts high-quality depth maps. Further, the proposed dense tracking scheme bridges camera pose estimation and dense 3D reconstruction by tracking against the global 3D model created with TSDF fusion. The quantita- tive and qualitative experiments show that TANDEM achieves better results than other state-of-the- art methods for both 3D reconstruction and visual odometry on synthetic and real-world data. We believe that TANDEM further bridges the gap between RGB-D mapping and monocular mapping.

---

Acknowledgments
We thank the anonymous reviewers for providing helpful comments that improved the paper. We express our appreciation to our colleagues, who have supported us, specifically we thank Mariia Gladkova (Technical University of Munich) and Simon Klenk (Technical University of Munich) for proof reading, as well as, Stefan Leutenegger (Technical University Munich & Imperial Col- lege London) and Sotiris Papatheodorou (Imperial College London) for their help with setting up demo for the conference. We thank the authors of iMAP, specifically Edgar Sucar (Imperial College London), for providing with evaluation data and scripts from their paper.
References
[1] R. A. Newcombe, A. Fitzgibbon, S. Izadi, O. Hilliges, D. Molyneaux, D. Kim, A. J. Davison, Kohi, J. Shotton, and S. Hodges. KinectFusion: Real-time dense surface mapping and tracking. In IEEE International Symposium on Mixed and Augmented Reality, 2011.
[2] C. Kerl, Sturm, and D. Cremers. Dense visual SLAM for RGB-D cameras. In IEEE Inter- national Conference on Intelligent Robots and Systems (IROS), 2013.
[3] T. Whelan, S. Leutenegger, R. Moreno, B. Glocker, and A. Davison. ElasticFusion: Dense SLAM without pose graph. In Robotics: Science and Systems (RSS), 2015.
[4] Zhang and S. Singh. Visual-lidar odometry and mapping: low-drift, robust, and fast. In IEEE International Conference on Robotics and Automation (ICRA), 2015.
[5] K. Tateno, Tombari. I. Laina, and N. Navab. CNN-SLAM: Real-time dense monocular SLAM with learned depth prediction. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[6] M. Bloesch, J. Czamowski, R. Clark, S. Leutenegger, and A. J. Davison. CodeSLAM learn- ing compact, optimisable representation for dense visual SLAM. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[7] J. Czarnowski, T. Laidlow, R. Clark, and A. J. Davison. DeepFactors: Real-time probabilistic dense monocular SLAMM IEEE Robotics and Automation Letters, 2020.
[8] X. Zuo, N. Merrill, W. Li, Y. Liu, M. Pollefeys, and G. Huang. C1de Visual-inertial odometry with learned optimizable dense depth. arXiv preprint arXiv:2012.10133: 2020.
[9] Z. Murez, T. van As, J. Bartolozzi, A. Sinha, V. Badrinarayanan, and A. Rabinovich. Atlas: End-to-end scene reconstruction from posed images. In European Conference on Computer Vision (ECCV), 2020.
[10] C. Tang and P. Tan. BA-Net: Dense bundle adjustment networks. In International Conference on Learning Representations (ICLR), 2019.
[11] Straub, T. Whelan, L. Ma, Y. Chen, E. Wijmans, S. Green, J. J. Engel, R. Mur-Artal, C. Ren, S. Verma, A. Clarkson, M. Yan, B. Budge, Y. Yan, X. Pan, J. Yon, Y. Zou, K. Leon, N. Carter, J. Briales, T. Gillingham, E. Mueggler, L. Pesqueira, M. Savva, D. Batra, H. M. Strasdat, R. D. Nardi, M. Goesele, S. Lovegrove, and R. Newcombe. The Replica dataset: A digital replica of indoor spaces. arXiv preprint arXiv:1906.05770 2019.
[12] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Sieg- wart. The euroc micro aerial vehicle datasets. The International Journal of Robotics Research, 15(10):1157-116 2016.
[13] A. Handa, Т. Whelan, J. McDonald, and A. J. Davison. A benchmark for RGB-D visual odometry, 3D reconstruction and SLAM. In IEEE International Conference on Robotics and Automation (ICRA), 2014.
[14] Y. Furukawa and J. Ponce. Accurate dense, and robust multiview stereopsis. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 2010.

---

[15] J. Stühmer, S. Gumhold, and D. Cremers. Real-time dense geometry from a handheld camera. In Pattern Recognition 33ndd DAGM Symposium, 2010.
[16] M. Pizzoli, C. Forster, and D. Scaramuzza. REMODE: Probabilistic, monocular dense recon- struction in real time. In IEEE International Conference Robotics and Automation (ICRA), 2014.
[17] J. L. Schönberger, E. Zheng, J.-M. Frahm, and M. Pollefeys. Pixelwise view selection for unstructured view stereo. In European Conference on Computer Vision (ECCV), 2016.
[18] S. Weder, J. L. Schönberger, M. Pollefeys, and M. R. Oswald. RoutedFusion: Learning real- time depth map fusion. In /EEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
[19] M. Ji, J. Gall, H. Zheng, Y. Liu, and L. Fang. SurfaceNet: An end-to-end 3D neural network for multiview stereopsis. In IEEE International Conference on Computer Vision (ICCV), 2017.
[20] A. Kar, C. Häne, and J. Malik. Learning a multi-view view stereo machine. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[21] R. Chen, S. Han, J. Xu, and H. Su. Point-based multi- view stereo network. In IEEE Interna- tional Conference on Computer Vision (ICCV), 2019.
[22] P.-H. Huang, K. Matzen, J. Kopf, N. Ahuja, and J.-B. Huang. DeepMVS: Learning multi-view stereopsis. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[23] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan. MVSNet: Depth inference for unstructured multi-vies villi-view stereo. In European Conference on Computer Vision (ECCV), 2018.
[24] H. Zhou, B. Ummenhofer, and T. Brox. DeepTAM: Deep tracking and mapping. In European Conference Computer Vision (ECCV), 2018.
[25] Y. Yao. Z. Luo, Li, T. Shen, T. Fang, and Quan. Recurrent MVSNet for high-resolution multi-view stereo depth inference. In IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), 2019.
[26] X. Gu, Z. Fan, S. Zhu, Z. Dai, F. Tan, and P. Tan. Cascade cost volume for high-resolution multi-view stereo and stereo matching. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[27] H. Yi, Z. Wei, M. Ding, R. Zhang, Y. Chen, G. Wang, and Y. Tai. Pyramid multi-view stereo with self-adaptive view aggregation. In European Conference on Computer Vision (ECCV), 2020.
[28] M. Niessner, M. Zollhöfer, S. Izadi, and M. Stamminger. Real-time 3D reconstruction at scale using voxel hashing. ACM Transactions on Graphics (TOG), 2013.
[29] F. Steinbrücker, J. Sturm, and D. Cremers. Volumetric 3D mapping in real-time on a CPU. In ICRA, pages 2021-2028. IEEE, 2014.
[30] E. Bylow, Sturm, C. Kerl, F. Kahl, and D. Cremers. Real-time camera tracking and 3D reconstruction using signed distance functions. In Robotics: Science and Systems (RSS), 2013.
[31] C. Kerl, Sturm, and D. Cremers. Robust odometry estimation for RGB-D cameras. In IEEE International Conference on Robotics and Automation (ICRA), 2013.
[32] O. Kähler, V. A. Prisacariu, and D. W. Murray. Real-time large-scale dense 3D reconstruction with loop closure. In European Conference on Computer Vision (ECCV), 2016.
[33] T. Schöps, T. Sattler, and M. Pollefeys. BAD SLAM: Bundle adjusted direct RGB-D SLAM. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
[34] E. Sucar, Liu, J. Ortiz, and A. Davison. iMAP: Implicit mapping and positioning in real-
time. In IEEE International Conference on Computer Vision (ICCV), 2021.
10

---

[35] R. A. Newcombe, S. J. Lovegrove, and A. J. Davison. DTAM: Dense tracking and mapping in real-time. In IEEE International Conference on Computer Vision (ICCV), 2011.
[36] J. Engel, T. Schöps, and D. Cremers. LSD-SLAM: large-scale direct monocular SLAM. In European Conference on Computer Vision (ECCV), 2014.
[37] Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 2018.
[38] R. Mur-Artal and J. D. Tardos. Probabilistic semi-dense mapping from highly accurate feature- based monocular SLAM. In Robotics: Science and Systems (RSS), 2015.
[39] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. ORB-SLAM: versatile and accurate monocular SLAM system. IEEE Transactions on Robotics, 2015.
[40] T. Schöps, T. Sattler, C. Häne, and M. Pollefeys. Large-scale outdoor 3D reconstruction mobile device. Computer Vision and Image Understanding, 2017.
[41] N. Yang, R. Wang. J. Stuckler, and D. Cremers. Deep virtual stereo odometry: Leveraging deep depth prediction for monocular direct sparse odometry. European Conference on Computer Vision (ECCV), 2018.
[42] N. Yang. L. von Stumberg, R. Wang, and D. Cremers. D3VO: Deep depth, deep pose and deep uncertainty for monocular visual odometry. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[43] T. Zhou, M. Brown, N. Snavely, and D. G. Lowe. Unsupervised learning of depth and ego- motion from video. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[44] Z. Yin and J. Shi. GeoNet: Unsupervised learning of dense depth, optical flow and camera pose. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
[45] H. Zhan, R. Garg, C. S. Weerasekera, K. Li, H. Agarwal, and I. M. Reid. Unsupervised learning of monocular depth estimation and visual odometry with deep feature reconstruction. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2018.
[46] K. M. Jatavallabhula, G. Tyer, and L. Paull. SLAM: Dense SLAM meets automatic dif- ferentiation. In IEEE International Conference on Robotics and Automation (ICRA), pages 2130-2137, 2020.
[47] B. Curless and M. Levoy. A volumetric method for building complex models from range images. In Conference Computer Graphics and Interactive Techniques (SIGGRAPH), 1996.
[48] Y. Furukawa and C. Hernández. Multi stereo: A tutorial. Foundations and Trends@ in Computer Graphics and Vision, 9(1-2)11-148, 2015.
[49] M. Jaderberg, K. Simonyan, A. Zisserman, and Kavukcuoglu. Spatial transformer networks. In Conference on Neural Information Processing Systems (NeurIPS). 2015.
[50] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Te- jani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance deep learning library. In Conference on Neural Information Process- ing Systems (NeurIPS), 2019.
[51] J. Park, P. Florence, J. Straub, R. Newcombe, and S. Lovegrove. DeepSDF: Learning con- tinuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 165-174, 2019.
[52] L. Mescheder, M. Oechsle, M. Niemeyer, S. Nowozin, and A. Geiger. Occupancy networks: Learning 3D reconstruction in function space. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 4460-4470, 2019.
11

---

[53] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. ScanNet: Richly- annotated 3D reconstructions of indoor scenes. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[54] D. Eigen, C. Puhrsch, and R. Fergus. Depth map prediction from a single image using multi- scale deep network. In Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger, editors, Conference Neural Information Processing Systems (NeurIPS), pages 1366-23744 Curran Associates, 2014.
12


Reviews:
[{"review_id": "q9hFFMFbbW", "reviewer": "Reviewer_W394", "summary": "This paper presented a new real-time dense SLAM system and achieves SOTA performance in both camera pose estimation and 3D reconstructions. It is built based on a combination of direct sparse odometry (DSO) and deep multi-view stereo network. It shows very good quantitative and qualitative results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written in general and easy to follow. It shows impressive results on camera pose estimation and 3D reconstruction and demonstrates good generalization results in unseen environments. The theoretical innovation is based on a combination of existing work. Nevertheless, the authors showed a well-designed system and explained the idea behind it. \n\nIt will be better if some minor issues that are mentioned in the issue section could be addressed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The presented paper showed very good results and contains sufficient engineering efforts to make it robust and run in real-time. ", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "4: The work contains interesting new ideas that will potentially have major impact in robotics or machine learning.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "* Evaluation in the 3D reconstruction: as the authors mentioned in Section 4.1 and 4.4, ScanNet dataset contains more artefacts and holes than Replica dataset. The proposed method was finetuned on the Replica dataset while the compared methods, i.e. DeepFactors, CNN-SLAM, Atlas were all trained on the ScanNet only. It will be a more fair comparison if the paper can also show the performance trained only on the ScanNet in order to remove the effect of the training dataset.\n* Evaluation in the camera pose estimation: the front-end tracking component in the compared VO/SLAM methods do not contain much about the learning component as the main innovation in the deepfactor is about its joint optimisation capability. Did the author try comparing with the more recent learning-based VO/SLAM system, such as the [25], and [43] that are mentioned in its related work section? \n\n"}}, {"review_id": "PbG-Dh6tNs-", "reviewer": "Reviewer_XeDS", "summary": "The paper presents an approach for dense tracking and mapping. Based on DSO, it performs photometric bundle adjustment to estimate the pose of keyframes in a sliding window. Poses and images are then forward to a cost-volume based learned module that returns the depth for every keyframe. The depth are then used to create a global TSDF-based representation of the environment. From this, depths a rendered to improve camera tracking in the front end.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The overall systems seems well engineered. It draws on state-of-the-art methods and the presented results are competitive.\n\nFrom a SLAM perspective, there is only a minor change when comparing to DSO. This is the use of the rendered depth (from the TSDF) for the front-end tracking. However, looking at the results, this adaptation seems to have a relevant effect on the overall system. This may be due to the increased image region that can be tracked against. It would be good to have a discussion on this somewhere.\n\nIn terms of reconstruction, the contribution is fairly incremental and mainly lies in combining prior work. This lies in adapting Cas-MVSNet to a visual aggregation module, and to merge all estimated depths into a TSDF.\n\nOne interesting aspect is the applicability of the paper to other data than the one it was trained on. On the SLAM side this could just mean that the tracking would fall back to DSO only, if the predicted depth become to inaccurate (some experiments to check this would be great).\n\nThe structure of the paper is good and the literature review appears complete.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "While the contribution is fairly incremental, the overall system seems to be performing well and both the increased performance of the front-end and the applicability of the approach to data it was not trained on is interesting.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "* The improved performance due to the rendered depth maps could be investigated more.\n* The order of tables and figures could better match the text", "reviewer_expertise": "Good: General knowledge of the area"}, "extra_text": {}}, {"review_id": "FZou1Gzdtvo", "reviewer": "Reviewer_xCmq", "summary": "This paper proposes a new dense SLAM system. The key novelty lies in the system design - a classical direct VO method is used for the tracking and a learned multi-view stereo (MVS) module is used for the dense mapping. This combination improves both the tracking accuracy (as the VO has access to dense depth maps) and the mapping accuracy as learned MVS is known to be very accurate. The results presented in the paper support this hypothesis.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper is clearly written and the method is well explained. The figures and the writing are polished and of publication quality.\n- The system design is novel in the sense that although multi-view stereo is closely related to dense SLAM, these approaches have largely been developing independently and have not been utilized for realtime SLAM. This is mainly because they used to be really computationally intensive, however, with the advent of deep-learning based multi-view stereo, such as MVS-Net these approaches have been much more efficient and thus utilising them in SLAM is a natural choice.\n- The evaluation in the paper is solid. The approach is compared to a number of recent systems allowing the reader to get a sense of its performance.\n- The results show that the system is accurate in both tracking and mapping. It achieves SOTA in terms of tracking and mapping accuracy on challenging datasets such as EuRoC. \n\n**Weaknesses**\n- There doesn’t seem to be any explanation of how the depth is initialized. This is quite important as the tracking relies on the depth rendered from the TSDF model, but at the start no model would have been built. Initialization is a very important problem in SLAM and I would expect some discussion on this point in the paper.\n- I can’t see a comparative evaluation of the runtime performance. There are some times reported for individual components, but what is the overall performance compared to other systems? I assume this is realtime as the approach is compared to other realtime systems in the evaluation. However, I think it is important that the authors include an overall runtime comparison. This is because most SLAM applications need realtime performance.\n- There are no examples of failure cases or challenging examples, like rotation-only motion. This is important as the system surely fails in certain situations and it would be good to know what these are.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "If the weaknesses above can satisfactorily be addressed, I recommend accepting the paper. The main reason is that I think it is a well-designed, clean system that achieves outstanding performance compared to the current state of the art.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "I expect the reviewers to address the issues above.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {}}, {"review_id": "BLy-vNd102", "reviewer": "Reviewer_mvF2", "summary": "The paper addresses the problem of visual odometry, dense depth estimation, and volumetric mapping with a monocular camera. The proposed method uses a direct sparse windowed optimization back-end and dense tracking front-end for visual odometry; CVA-MVSNet based on multi-view stereo and a neural network architecture for dense depth estimation; and a truncated signed distance function to represent the 3D map. Experiments are performed to evaluate both the quality of visual odometry and 3D reconstruction on public datasets.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes an interesting integration of a traditional visual SLAM pipeline with a learning-based approach for 3D reconstruction, displaying positive results compared to other methods. The experiments are quite complete as different parts of the pipeline are evaluated and an ablation study is included.\n\nOne of the main comments regards the fact that a dense SLAM system is proposed, that eventually could be deployed on a robot, however currently it has been tested on a powerful workstation, limiting the potential applicability on mobile robots. The paper could discuss about deployments on embedded systems.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, while the paper presents a number of comments, including that it could be more explicit on the possibility to deploy the system on embedded systems, the paper presents a technically sound integration that shows an improvement with respect to other methods in visual SLAM and 3D reconstruction with a monocular camera.", "technical_quality": "Very Good:  Convincing.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "Very good: Comprehensive knowledge of the area"}, "extra_text": {"issues": "Other specific comments:\n- \"... over RGB-D [1] or LiDAR-based [4] solutions: They are not limited to indoor environments, they are not limited to near-range sensing and they...\" This seems to imply that, e.g., LiDAR is limited to near-range sensing, when LiDAR can get to up to in the order of 100m, when the camera is much more limited. Please rephrase. \n- The related work section could tie more with the proposed work; for example, in 3D reconstruction, the link is explicit; the link is not explicit for monocular SLAM.\n- to have a more complete evaluation of the visual odometry, feature-based SLAM system, as ORB-SLAM could be included in the comparison.\n- justification of the results are important to provide, e.g., why CVA-MVSNet achieves better results with the model trained on the synthetic Replica?\n\nMinor typos/presentation problems:\n- page 2, line 58, \"to weight\" -> \"to weigh\"\n- page 6, line 213 \"TANDEM is a dense SLAM system that estimates camera poses in real-time.\", this is a repetition -- at this point TANDEM has been already introduced multiple times, thus the sentence can be deleted.\n- Table 5 appears better if not at the end of the page."}}]

Author rebuttal:

Meta-review: The paper studies an interesting problem and provides a well-engineered solution. The reviewers appreciated the technical approach, the clarity of the presentation, the experimental evaluation, and the applicability of the approach outside the training regime. 
The main weaknesses are:
- The approach is incremental and relies on a small modification of DSO and Cas-MVSNet. However, the resulting system provides a compelling integration of classical VO and deep MVS, which can trigger further work from the community.
- The computational cost of the approach might prevent applications on an actual robot / embedded system. After the rebuttal, the authors added further discussion about potential runtime optimization in the supplementary material, which the reviewers and area chair found satisfactory.
- The presentation in the original submission lacked details about the initialization of the TSDF, a discussion about runtime, choice of training/testing datasets, and a comparison against more recent VO/SLAM systems. These shortcomings have been resolved in the revised paper.

Decision: Accept (Poster)