DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
Authors: Zachary Teed
Jia Deng
Venue: NeurIPS 2021 Oral (2021)
Keywords: SLAM
Simultaneous Localization and Mapping
3D

Abstract: We introduce DROID-SLAM, a new deep learning based SLAM system. DROID-SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through a Dense Bundle Adjustment layer. DROID-SLAM is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training on monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https://github.com/princeton-vl/DROID-SLAM.

DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
Zachary Teed
Jia Deng Princeton University
jadee//djooaee edu
Abstract
We introduce DROID-SLAR a new deep learning based SLAM system. DROID- SLAM consists of recurrent iterative updates of camera pose and pixelwise depth through Dense Bundle Adjustment layer. DROIDD-LLAN is accurate, achieving large improvements over prior work, and robust, suffering from substantially fewer catastrophic failures. Despite training monocular video, it can leverage stereo or RGB-D video to achieve improved performance at test time. The URL to our open source code is https::/ wttptth...om/prracttttao/////0/0
Introduction
Simultaneous Localization and Mapping (SLAM) aims to (1) build a map of the environment and (2) localize the agent within the environment. It is special form of pnncturr-from-Motiio (SfM) focused on accurate tracking of long-term trajectories. is a critical capability for robotics, especially autonomous vehicles. In this work, address visual SLAM, where sensor recordings come in the form of images captured from monocular, stereo, or RGB-D camera.
The SLAM problem has been approached from number of different angles. Early work was built using probabilistic and filtering based approaches [2 30,, and alternating optimization of the map and camera poses 334 [6]. More recently, modern SLAM systems have leveraged least-squares optimization. A key element for accuracy has been full Bundle Adjustment (BA), which jointly optimizes the camera poses and the 3D map in single optimization problem. One advantage of the optimization- based formulation is that SLAM system can be easily modified to leverage different sensors. For example, ORB- RRB-S (3 supports monocular, stereo, RGB-D, and IMU sensors, and modern systems can support a variety of camera models 5 27 42 (6. Despite significant progress, current SLAM systems lack the robustness demanded for many real-world applications. Failures come in many forms, such as lost feature tracks, divergence in the optimization algorithm, and accumulation of drift.
Deep learning has been proposed as solution to many of these failure cases. Previous work has investigated replacing hand-crafted with learned features [3 7 29 26) 33), using neural 3D representations 46
9
22), and combining learned energy terms with classical op- 57 Other work has tried to lear SLAM or VO systems end-to-end
45
timization backends 58
While these systems are sometimes more robust, they fall far short of the accuracy of their classical counterparts common benchmarks.
59
47
53
52
46).
In this work we introduce DROID-SLAM, a new SLAM system based on deep learning. It has state-of-the-art performance, outperforming existing SLAM systems, classical or learning-based, on challenging benchmarks with very large margins. In particular, it has the following advantages:
We achieve large improvements over prior work across multiple datasets and modalities. On the TartanAir SLAM competition (54], we reduce error by 62% over the best prior result on the monocular track and 60% on the stereo track. We rank 1st on the
High Accuracy:
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

SSAID-SLAM
Monocular, Stereo or RGB-D Video
Apose
Adepth
Figure I: DROID-SLAM can operate on monocular, stereo, and RGB-D video. It builds a dense 3D map of the environment while simultaneously localizing the camera within the map.
ETH-3D RGB-D SLAM leaderboard 41, outperforming the second place by 35% under the AUC metric which considers both eπror and rate of catastrophic failure. On EuRoC [2]. with monocular input, we reduce error by 82% among methods with zero failures, and by 43% over ORB-SLAM3 considering only the 10 out of 1 sequences it succeeds on. With stereo input, we reduce error by 71% over ORB-SLAME On TUM-RGBD 43 we reduce error by 83% among the methods with zero failures.
High Robustness: We have substantially fewer catastrophic failures than prior systems. On ETH-3D, we successfully track 30 of the 32 RGB-D B datasets, while the next best successfully tracks only 19/32. On Air,anAir, EuRoC, and TUM-RGBD, we have zero failures.
Strong Generalization: Our system, trained only with monocular input, can directly use stereo or RGB-D input to get improved accuracy without any retraining. All of our results across 4 datasets and 3 modalities are achieved by single model, trained once with only monocular input entirely on the synthetic TartanAir dataset.
The strong performance and generalization of DROID-SLAM is made possible by its "Differentiable Recurrent AptimizationnInspire Design" (DROID), which is an end-to-end differentiable architecture that combines the strengths of both classical approaches and deep networks. Specifically, consists of recurrent iterative updates, building upon RAFT 48 for optical flow but introducing two key innovations.
First, unlike RAFT, which iteratively updates optical flow, we iteratively update camera poses and depth. Whereas RAFT operates on two frames, our updates are applied an arbitrary number of frames, enabling joint global refinement of all camera poses and depth maps, essential for minimizing drift for long trajectories and loop closures.
Second, each update of camera poses and depth maps in DROID-SLAM is produced by a differentiable Dense Bundle Adjustment (DBA) layer, which computes Newss- update camera poses and dense per-pixel depth so as to maximize their compatibility with the current estimate of optical flow. This DBA layer leverages geometric constraints, improves accuracy and robustness, and enables a monocular system to handle stereo or RGB-D input without retraining.
The design of DROID-SLAM is novel. The closest prior deep architectures Deep [47 and BA-Net [46), both of which were focused on depth estimation and reported limited SLAM V2D alternates between updating depth and updating camera poses, instead of bundle adjustment. BA-Net has a bundle adjustment layer, but their layer is substantially different: is not "dense" in that it optimizes over small number of coefficients used to linearly combine depth basis (a set of pre-predicted depth maps), whereas we optimize over per-pixel depth directly, without being handicapped by depth basis. In addition, BA-Net optimizes photometric reprojection error (in feature space), whereas we optimize geometric error, leveraging state-of-the-an flow estimation.
results. V2D
We perform extensive evaluation across four different datasets and three different sensor modalities, demonstrating satte-of-hheaane performance in all cases. We also include ablation studies that shed light on important design decisions and hyperparameters.
2
Related Work
Modern SLAM systems treat localization and mapping joint optimization problem 4

---

Visual SLAM focuses on observations in the form of monocular, stereo, or RGB-D images. These approaches are commonly categorized as either being direct or indirect [13]. Indirect approaches [3] first process the image into an intermediate representation by detecting points of interest and attaching feature descriptors. Features are then matched between images. Indirect approaches optimize camera pose and a 3D point cloud by minimizing reprojection error-the distance between a projected 3D point and its location in the image.
32 S 37
Direct approaches model the image formation process and define an objective function over photomet- ric error T6 15 60 One advantage of direct approaches is that they can model more information about the image, such as lines and intensity variations [15] which are not used by indirect approaches. However, photometric errors typically lead to more difficult optimization problems, and direct ap- proaches are less robust to geometric distortion such as rolling shutter artifacts. This approach requires more sophisticated optimization techniques, such as coarse-to- image pyramids to avoid local minimum.
Our method does not clearly fit into either of the categories. Like the direct approach, we do not require preprocessing steps to detect and match features between the images. We instead use the full image, allowing us leverage wider range of information than indirect methods with typically only use corners and edges. However, we minimize reprojection error similar to indirect methods. This an easier optimization problem and avoids the need for more complicated representations such as image pyramids. In this sense, our approach borrows the best of both approaches: the smoother objective function of indirect approaches with the greater modeling capacity of indirect approaches.
Deep Learning has more recently been applied to the SLAMM problem. Many works have focused
on training systems for partieular subproblems, such feature detection [13 7 29
26 35 feature
matching and outlier rejection [38, (36]. and localization [5] 39], SuperGlue (38] was designed to perform feature matching and verification and make 2-view pose estimate much more robust. Our network also draws inspiration from Dusmanu et al [1]. which builds a neural network into the SfM pipeline to improve keypoint localization accuracy.
53 These methods not full SLAM systems, but instead focus small scale reconstruction on the order of two 8 (50 53 up to a dozen frames [59] 46, 47). They lack many of the core capabilities of modern SLAM systems such loop closure and global bundle adjustment which inhibit their ability to perform large scale reconstruction as demonstrated
Other works have focused on training SLAM systems end-to-end [59]
our experiments. VSLAM implements several existing SLAM algorithms as differentiable computation graphs, allowing for errors in the reconstruction to be backpropagated back to sensor measurements. While this approach differentiable, it has no trainable parameters, meaning the performance of the system limited by the accuracy of the classical algorithm they emulate.
DeepFactors 9 is the most complete deep SLAM system, building on the earlier CodeSLAM It performs joint optimization of the pose and depth variables, and is capable of short and long long-range loop closure. Similar to BA-Net 46), DeepFactors optimizes the parameters of learned depth basis during inference. In contrast, we do not rely on a learned basis, but instead optimize pixelwise depth. This allows our network to better generalize to new datasets since depth representation is not tied to the training dataset.
Approach
We take video input with two objectives: estimate the trajectory of the camera and build 3D map of the environment. We first describe the monocular setting; in Sec. 3.4 we describe how to generalize the system to stereo and RGB-D video.
Representation: Our network operates on an ordered collection of images, { For each image t, we maintain two state variables: camera pose G; E (3) and inverse depth d, E R # The set of poses, {G, N and set of inverse depths {d, N are unknown state variables, which get iteratively updated during inference new frames processed. For the reminder of the paper, when we refer to depths, note that we are using the inverse depth parameterization.
We adopt frame-graph (V.E) to represent co-visibility between frames. An edge (i,j) E means image I, and I, have overlapping fields of view which shared points. The frame graph built dynamically during training and inference. After each pose depth update, we can recompute
3

---

C
h
.00.0
CunvGRU
ComGRU
11 T in. 43
E
G
Figure 2: Illustration of the update operator: The operator acts edges in the frame graph, predicting flow revisions which are mapped to depth and pose update through the (DBA) layer.
visibility to update the frame graph. If the camera returns to a previously mapped region, we add long range connections in the graph perform loop closure.
3.1 Feature Extraction and Correlation
Features are extracted from each new image added to they system. Key components of this stage are borrowed from RAFT[48]
Feature Extraction Each the input images are processed by a feature extraction network. The network consists of 6 residual blocks and downsampling layers, producing dense feature maps at the input image resolution. Like 4AFT[SS] we use two separate networks: a feature network and context network. The feature network is used to build the set of correlation volumes, while the context features are injected into the network during each application of the update operator.
Correlation Pyramid For each edge in the frame graph, (i.j) E E, we compute 4D correlation volume by taking the dot product between all all-pairs of feature vectors in go (I,) and go (I,))
Ci M2giiag22
(ge( (e)())ar 9o (12)2
We then perform average pooling of the last two dimension of the correlation volume following RAFT 48 to form a 4-level correlation pyramid.
Correlation Lookup We define a lookup operator which indexes the correlation volume using grid with radius r, L, H x xH x #xxxx2 + R2 [XXxWx(++))
The lookup operator takes an H x W grid of coordinates input and values are retrieved from the correlation volume using bilinear interpolation. The operator is applied each correlation volume in the pyramid and the final feature vector is computed by concatenating the results at each level.
3.2 Update Operator
The core component of SLAM system is learned update operator show in Fig. The update operator is 3 x 3 convolutional GRU with hidden state h. Each application of the operator updates the hidden state, and additionally produces pose update, (k and depth update, (*) The pose and depth updates are applied to the current depth and pose estimates through retraction on the SE3 manifold and vector addition respectively
(k+1) Exp(Ag(() (A).
(k+1)
(k) (R)
(2)
Iterative applications of the update operator produce sequence of poses and depths, with the expec- tation of converging fixed point (G()) + G*, (d(*) d reflecting the true reconstruction.
Correspondence At the start of cach iteration we use the current estimates of poses and depths to estimate correspondence. Given grid of pixel coordinates, p: 1 #xXx2 in frame i, compute the dense correspondence field pij
Pú (Gi 11,"" (p. (p..d))),
Púj Xxxx2
Gūj = G, o G.
(3)

---

for each edge (i.j) E & in the frame graph. Here II. is the camera model mapping set of 3D points onto the image and II_", is the inverse projection function mapping inverse depth map d and coordinate grid p. to 3D point cloud (we provide formulas and Jacobians in the appendix). Pij represents the coordinates of pixels p. mapped into frame j using the estimated pose and depth.
Inputs We use the correspondence field to index the correlation volumes. For each edge (i, E E we use Pij perform lookup from the correlation volume Cij to retrieve correlation features. Additionally, we use the correspondence field to derive optical flow induced by camera motion as the difference Pıj Pj. Furthermore, the residual from the previous BA solution is concatenated with the flow field allowing the network to use feedback from the previous iteration.
The correlation features provide information about visual similarity in the neighbourhood of Pij allowing the network to learn to align visually similar image regions. However, correspondence is sometimes ambiguous. The flow provides complementary source of information allowing the network to exploit smoothness in the motion fields to gain robustness.
Update The correlation features and flow features are each mapped through two convolutional layers before being injected into the GRU. Additionally, we inject context features, as extracted by the context network, into the GRU through element-wise addition.
The ConvGRU is local operation with small receptive field. We extract global context by averaging the hidden state across the spatial dimensions of the image and use this feature vector as additional input to the GRU. Global context is important SLAM because incorrect correspondences, caused by large moving objects for example, can degrade the accuracy of the system. It is important for the network to recognize and reject erroneous correspondence.
The GRU produces updated hidden state h (k+1), Instead of predicting updates to the depth pose directly, we instead predict updates in the space of dense flow fields. We map the hidden state through two additional convoluation layers to produce two outputs: (1) revision flow field Faj E #fxxxx2 x2 and (2) associated confidence map Waj E R+ x2 The revision Fay is a correction term predicted by the network to correct errors in the dense correspondence field. We denote the corrected correspondence p.j Fij + Pij
We then pool the hidden state over all features which share the same source view i and predict a pixel-wise damping factor X. We use the softplus operator to ensure that the damping term is positive, Additionally, we use the pooled features to predict a 8x8 mask which can used to upsample the inverse depth estimate.
Dense Bundle Adjustment Layer (DBA) The Dense Bundle Adjustment Layer (DBA) maps the set of flow revisions into a set of pose and pixelwise depth updates. We define the cost function over the entire frame graph
E(G', d') El (G o (p, I.
Eg diag Waj-
(4)
(E,FEE
where |1=1ls: is the Mahalanobis distance which weights the error terms based the confidence weights Way- Eqn. 4 states that we want an updated pose G' and depth d' such that reprojected points match the revised correspondence př, predicted by the update operator.
We use local parameterization linearize Eqn. d and use the Gauss-Newton algorithm solve for updates (Ag. Ad). Since each term in Eqn. 4 only includes a single depth variable, the Hessian matrix has block diagonal structure. Separating pose and depth variables, the system can solved efficiently using the Schur complement with the pixelwise damping factor À added to the depth block
B ET
E
A Ad
A
B EC 'ET -v (1-------- EEC''w) 'w)
C
w
Ad
C- w ET
where C is diagonal and can be cheaply inverted C-1 1/C. The DBA layer is implemented as part of the computation graph and backpropogation is performed through the layer during training.
3.3 Training
Our SL.AM system is implemented in PyTorch and we use the Lie Torch extension 49 to perform backprogation in the tangent space of all group elements.
5

---

Removing gauge freedom In the monocular setting, the network is only able to recover the trajectory of the camera up to similarity transform. One solution is to define loss which is invariant to similarity transforms. However, the gauge-freedde nruge-freedom still exists during training which poorly impacts the conditioning of the linear system and the stability of the gradients. We solve this problem by fixing the first two poses to the ground-truth poses of each training sequence. Fixing the first pose removes the 6-dof gauge freedom. Fixing the second pose resolves the scale freedom.
Constructing training video Each training example consists of a 7-frame video sequence. In order to ensure stable training and good downstream performance, we want to sample videos which are not too easy nor too difficult.
The training set is composed of collection of videos. For each video i of length No, we precompute an N; x Nr distance matrix storing the average optical flow magnitude between each pair of frames. However, not all frames are covisible; and frames pairs with less than 50% overlap are assigned a distance of infinity. During training, we dynamically generate videos by sampling paths in the distance matrix, such that the average flow between adjacent video frames is between 8px and 96px.
Supervision We supervise our network using a combination of pose loss and flow loss. The flow loss is applied to pairs of adjacent frames. We compute the optical flow induced by the predicted depth and poses and the flow induced by the ground truth depth and poses. The loss is taken to be the average 12 distance between the two flow fields.
Given a set of ground truth poses (T) N and predicted poses (G) N the pose loss is taken to be the distance between the ground truth and predicted poses, frose = Σ Logsva (T. G. We apply the losses to the output of every iteration with exponentially increasing weight using y = 0.9.
3.4
SLAM System
During inference, we compose the network into full SLAM system. The SLAM system takes a video stream as input, and performs reconstruction and localization in real-time. Our system contains two threads which run asynchronously. The frontend thread takes in new frames, extracts features, selects keyframes, and performs local bundle adjustment. The backend thread simultancously performs global bundle adjustment over the entire history of keyframes. We provide an overview of the system here, and provide more information in the appendix.
Initialization Initialization is simple with DROID-SLAM. We simply collect frames until we have a set of 12. As we accumulate frames, we only keep the previous frame if optical flow is greater than 16px (estimated by applying one update iteration). Once 12 frames have been accumulated, we initialize a frame graph by creating an edges between keyframes which are within 3 timesteps apart, then run 10 iterations of the update operator.
Frontend The frontend operates directly on the incoming video stream. It maintains collection of keyframes and a frame graph storing edges between covisible kefyrames. Keyframe poses and depths are actively being optimized. Features are first extracted from the incoming frames. The new frame is then added to the frame graph adding edges with its 3 closest neighbors as measured by mean optical flow. The pose is initialized using a linear motion model. We then apply several iterations of the update operator to update keyframe poses and depths. We fix the first two poses to remove gauge freedom but treat all depths as free variables.
After the new frame is tracked, we select a keyframe for removal. We compute distance between pairs of frames by computing the average optical flow magnitude and remove redundant frames. If no frame is good candidate for removal, we remove the oldest keyframe.
Backend The backend performs global bundle adjustment over the entire history of keyframes. During each iteration, we rebuild the frame graph using the flow between all pairs of keyframes, represented as an N x N distance matrix. We first add edges between temporally adjacent keyframes. We then sample new edges from the distance matrix in order of increasing flow. With each selected edge, we suppress neighboring edges within distance of 2, where distance is defined as the Chebyshev distance between index pairs [|(i,]) - (k. l) = max(li kl, Z)).
We then apply the update operator to the entire frame graph, often consisting of thousands of frames and edges. Storing the full set of correlation volumes would quickly exceed video memory. Instead, we use the memory efficient implementation proposed in RAFT 48).

---

eed
ELLAAA AARADANAA
Figure 3: DROID-SLAM can generalize to new datasets. In order, we show results from Tanks Temples [1). ScanNet [T0), Sintel (3. and ETH-3D 41: all using monocular video.
During training, we implement dense bundle adjustment in PyTorch to leverage the automatic differentiation engine. At inference time, of the block-sparse structure of the problem, then perform sparse Cholesky decomposition on the reduced camera block.
custom CUDA kernel which takes advantage
We only perform full bundle adjustment on keyframe images. In order to recover the poses of non-keyframes, we perform motion-only bundle adjustment by iteratively estimating flow between each keyframe and its neighboring non-keyframes. During testing, we evaluate on the full camera trajectory, not just keyframes.
Stereo and RGB-D Our system can easily modified for stereo and RGB-D video. the case of RGB-D, we still treat depth as variable, since sensor depth can be noisy and have missing observations, and simply add term to the optimization objective (Eqn. + which penalizes the squared distance between the measured and predicted depth. For stereo, we use the exact same system described above, with just double the frames, and fix the relative pose between the left and right frames in the DBA layer. Cross camera edges in the graph allow us to leverage stereo information.
Experiments
We experiment diverse of datasets and sensor modalities. We compare both deep learning and established classical SLAM algorithms and put specific emphasis on cross- cooss-dttaset generalization. Following prior work, we evaluate the accuracy of the camera trajectory [3] 41), primarily using Absolute Trajectory Error (ATE) [43]. While some datasets have ground truth point clouds [21), there no standard protocol compare 3D reconstructions directly given by SLAM systems because
SLAM systems can choose which 3D points to reconstruct. Evaluating dense 3D reconstruction is typically considered the domain of Multiview Stereo 19 and outside the scope of this work.
Our network is trained entirely on monocular video from the synthetic TartanAir dataset [54]. We train network for 250k steps with batch size of 4, resolution 384 512, and 7 frame clips, and unroll 15 update iterations. Training takes 1 week on 4 RTX-3090 GPUs.
Monocular
MIH000
MI100]
M1R002
MI3003
MI1004
MUDDSS
M11006
MH0007
DD
77
6.15
2.12
4,54
3.89
2.71
11.55
5.53
3.76
5.03
TortanVO
33
0.26
0.94
3.19
2.04
Ours
0.08
0.05
0.04
0.02
0.01
1.31
0.30
0.07
0.24
Table 1: Results on the TartanAir monocular benchmark.
7

---

TartanAir (54) (Monocular & Stereo) The TartanAir dataset is a challenging synthetic benchmark for evaluating SLAM algorithms and was used as part of the ECCV 2020 SLAM competition. We use the official test split [54]. and provide ATE across all "Hard" sequences in Tab. [
Tab. demonstrates both the robustness of our method (no catastrophic failures) and accuracy (very low drift). We retrain DeepV2D [47] on TartanAir as baseline. On most sequences, we outperform existing methods by an order-of-magnitud and achieve 8x lower average error than TartanVO (53) and 20x lower than V2D 47]. We also use the TartanAir dataset to compare with the top The top two submissions use systems
submissions to the ECCV 2020 SLAM competition Tab. 2
Mono.
Sterco
SVAM 17
0,510
0.182
VOLDOR
0.177
SuperGiuee 1S
SuperPoint
[R COLMAP 40
0,340
0.119
Ours
0.129
0.047
Table 2: Results on the TartanAir test set, compared with the top 3 submission to the ECCV 2020 SLAM competition. The score is computed using normalized relative pose error for all possible sequences of length (5, 10, 15, (0.,..00) meters, see competition page for details.
built on top of COLMAP 40 and run 40x slower than real-time. Our method, on the other hand, runs 16x faster and achieves error 62% lower on the monocular benchmark and 60% lower on the stereo benchmark.
EuRoC [2] (Monocular & Stereo) In the remaining experiments, we are interested in the ability of our network to generalize to new cameras and environments. The EuRoC dataset consists of video captured from sensor on- on-board micro acrial vehicle (MAV) and is widely used benchmark to evaluate SLAM systems. We use the EuRoC dataset to evaluate both monocular and stereo performance and report results on Tab. 3
MI01
MI802
MH303
MHH1
MH05
V10]
V102
V103
V201
V202
V203
Avg
Deepflyh.
DespFactors
0
1.587
1.479
3.139
5.351
4.002
1.5 1.500
01679
0.900
0.876
1.905
1,021
2,040
DeepY2D
+7
0.739
1.144
1.567
1,570
0.290
2.202
2.743
1.298
DcepV2D Taraa Airj
1614
1,922
16655
1.775
1.013
0.717
0.695
1.483
0.839
1.052
0.591
L.173
TortanVO
0.639
0.325
0.550
1.153
1.021
0.447
0.389
0.622
0.433
0.749
1.152
0.680
D3VO DSO 57
0.08
0.09
0.11
0.05
0.19
DRB SLAM 31
0,071
0.067
0.071
0.082
0.060
0.015
0.020
X
0.021
0018
X
Classical
0.016
0.046
3.810
0.110
1.152
0.601
SVO [
0,100
0,120
0.410
0.430
0.300
0.070
0.210
X
0.110
0.110
1 1,080
DSM [0
0,039
D.06
0.055
0.057
0.067
0.055
0.959
0.076
0.056
0.057
01774
0.136
ORB-SLAMM
0.016
0.027
0.072
0.033
0.029
Ouns (odometry only)
0.163
0.121
0.242
0.399
0.270
0.105
0.165
0.158
0.102
0.115
0.204
0.186
Ours
0,013
0.014
0.022
0.043
0.043
0.057
0.012
0.020
0.017
0.013
0.014
0.022
Table 3: Monocular SLAM on the EuRoC datasets, ATE[m].
denotes visual odometry methods.
In the monocular setting, we achieve an average ATE of 2.2cm, reducing error by 82% among methods with zero failures, and by 43% over ORB-SLAM3 when only comparing sequences where ORB-SLAN ORB-SLAM3 is successful.
We compare to several deep learning approaches. We compare to Dee trained on the TartanAir DeepFactors g perform poorly
dataset and the publicly available version trained on NYUv22 33 and ScanNet 10
was trained on ScanNet. We find that recent deep learning approaches g 47 53
on the FuRoC dataset compared to classical SL.AM systems. This is due to poor generalization and dataset biases which lead large amounts of drift: our method does not suffer from these issues. D3VO [57] is able to achieve both good robustness and accuracy by combining neural network frontend with DSO as a backend, using 6 of the 11 sequences for evaluation and performing unsupervised training on the remaining ones, which contain the same scenes used for evaluation.
The RGBD dataset consists of indoor scenes captured with handheld camera. This is a notoriously difficult dataset for monocular methods due to rolling shutter artifacts, motion blur, and heavy rotation. We benchmark prior work on the entirety of the freiburg] set in Tab.
TUM-RGBD 43
Classical SLAM algorithms such as ORB-SLAM tend to fail on most of the sequences. While deep learning methods are more robust, they obtain low accuracy most of the evaluated sequences. Our
8

---

floo
plane
room
FpV
soddy
xyz
agg
12
0.023
SRB-SLLAMS
0.210
DecpTAM
59
0.111
0.053
0.103
0.206
0.064
0.239
0.093
0.144
0.033
0.116
V
53
0.178
0.125
0.122
0.349
0.297
0.333
0.049
0.339
0,062
0.236
DespV2D
0.341
0.166
0.379
1.653
0.205
0.246
0.105
0.316
0.064
0.575
DeepV2D (TrranAi))
0.182
0.622
0.633
0.579
0.562
0.776
0.033
0.602
0.150
0,468
DespFactors
0.253
0.038
Table 4: ATE on the TUM-RGBD benchmark. All methods are provided mono. vidco, 'except DeepTAM which uses RGB-D and "Tartan which uses ground truth to scale relative pose.
method is both robust and accurate, successfully tracks all sequences while achieving 83% lower ATE than DeepFactors [9] and which succeeds all videos and 90% lower ATE than Deep 47)
FTH-3D 3D-33 St AM Benchmark
Method
(triin
AUC
BundieFusion
84.10
33.84
25
ElasticFrasion
56
89.06
34.02
20
RFusion
$1.94
DVO-SLAM
193.8
71.83
ORB-SLAM2
32
156.10
104.28
CRB SLAM2
BAD-SLAM
280.05
153.47
DVD-SLAM Bundlisuusion
Ours
340.42
207.79
ATE [cm]
Figure 4: Generalization results on the RGB-D WTH3D-SLAM benchmark. (Left) Our method, which is trained only on the synthetic TartanAir dataset, ranks 1st on both the train and test splits. (Right) Plot of the number successful trajectories as function of ATE. Our method successfully tracks 30/32 of the datasets where image data is available.
ETH3D-SLAM [41] (RGB-D) Finally, we evaluate the RGB-D performance on the ETH3D-SLAM benchmark. In this setup, the network is also provided measurements from RGB-D camera. We take our network trained on TartanAir and add an addition term in the optimization objective penalizing the distance between the predicted inv. depth and inv. depth measured by the sensor. Without any finetuning, our method ranks st on both the train and test splits. Several the datasets "dark" meaning image data is available; these datasets do not submit any predictions. On the test set, we successfully track 30/32 RGB-D, improving over the next best 19/32.
Timing and Memory Our system can in real-time with 2 3090 GPUs. Tracking and local BA is run on the first GPU, while global BA and loop closure is run on the second. On EuRoC, we average 20fps (camera hz) by downsampling 320 x 512 resolution and skipping every other frame. Results in Tab. were obtained in this setting. On TUM-RGBD we average 30fps by downsampling to 240 x 320 and skipping every other frame, again the reported results where obtained in this setting. On TirtanAir, due to much faster camera motion, we are unable to run in real-time time, averaging 8fps. However, this still 16x speedup over the top 2 submissions to the TartanAir SLAM challenge, which rely on COLMAP 40)
The SLAM frontend can be run on GPUs with 8GB of memory. The backend, which requires storing feature maps from the full set of images, is more memory intensive. All results on TUM-RGBD can be produced on single 1080Ti graphics card. Results on EuRoC, TartanAir and ETH-3D (where video can be up to 5000 frames) requires a GPU with 24GB memory. While memory and resource requirements currently the biggest limitation of our system, believe these can be drastically reduced by culling redundant computation and more efficient representations.
Conclusion
We introduce AROID-SLAND an end-to-end neural architecture for visual SLAM. DROID-SLAM accurate, robust, and versatile and can be used on monocular, stereo, and RGB-D video. outperforms prior work by large margins challenging benchmarks.
Acknowledgements This work is partially supported by the National Science Foundation under Award IIS- 194491.....

---

References
[1] M. Bloesch, J. Czarnowski, R. Clark, S. Leutenegger, and J. Davison. Codeslam-keariing a compact, optimisable representation for dense visual slam. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2560-2568, 2018.
[2] M. Burri, J. Nikolic, P. Gohl, T. Schneider, J. Rehder, S. Omari, M. W. Achtelik, and R. Siegwart. The euroc micro aerial vehicle datasets. The International Journal of Robotics Research, 1577:1111-1163 2016.
[3] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In Fitzgibbon al. (Eds.), editor, European Conf. on Computer Vision (ECCV), Part IV, LNCS 7577. pages 611-625 625. Springer- Verlag. Oct. 2012.
[4] C. Cadena, L. Carlone, H. Carrillo, Y. Latif, D. Scaramuzza, J. Neira, I. Reid, and J. J. Leonard. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics,
2016.
G. Rodríguez, J. M. Montiel, and D. Tardós. Orb-slam3: An accurate open-source library for visual, visual-incrtin and multi map slam. arXiv preprint arXiv:2007.1189h 2020. [6] D. Caruso, J. Engel, and D. Cremers. Large-scale direct slam for omnidirectional cameras. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 141-148. IEEE, 2015.
[5] C. Campos, R. Elvira,
[7] C. Choy, J. Gwak, S. Savarese, and M. Chandraker. Universal correspondence network. arXiv preprint arXiv:1606.03553 2016.
[8] R. Clark, M. Bloesch, Czarnowski, S. Leutenegger, and A. J. Davison. Ls-net: Learning to solve
nonlinear least squares for monocular stereo. arXiv preprint arXiv: 1809. /809, 02966, 2018.
[9] J. Czarnowski. T. Laidlow, R. Clark, and A. J. Davison. Deepfactors: Real-time probabilistic dense monocular slam. IEEE Robotics and Automation Letters, 5(2):721-728, 2020.
[10] A. Dai, A. X. Chang, M. Savva, M. Halber, T. Funkhouser, and M. Nießner. Scannet: Richly-annotated 3d
reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 5828-5839, 2017.
[11] A. Dai. M. Nießner, M. Zollhöfer, S. Izadi, and C. Theobalt. Bundlefusion: Real-time globally consistent
3d reconstruction using on- fly surface reintegration. ACM Transactions on Graphicx (ToG), 36(4):1. 2017.
[12] A. J. Davison, I. D. Reid, N. D. Molton, and O. Stasse. Monoslam: Real-time single camera slam. IEFE transactions on pattern analysis and machine intelligence, 29(6); 1052-1067 2007.
[13] D. DeTone, T. Malisiewicz, and A. Rabinovich. Superpoint: Self supervisd interest point detection and description. In Proceedings of the IEEE conference computer vision and paltemn recognition workshops, pages 224-236, 2018
[14] M. Dusmanu, J. L. Schonberger, and M. Pollefeys. Multi-view optimization of local feature geometry. In Proceedings of the 2020 European Conference on Computer Vision, 2020.
[15] J. Engel, V. Koltun, and D. Cremers. Direct sparse odometry. IEEE transactions pattern analysis and machine intelligence, 40(3):61 1 1-625, 2017.
[16] J. Engel. T. Schöps, and D. Cremers. Lsd-slam: Large-scale direct monocular slam. In European conference on computer vision, pages 834 849. Springer, 2014.
[17] M. Ferrera, A. Eudes, J. Moras, M. Sanfourche, and G. Le Besnerais. Ov2 slam: A fully online and versatile visual slam for real-time applications. IEEE Robotics and Automation Letters, 6(2):1399-1406,
2021.
[18] C. Forster, Z. Zhang, M. Gassner, M. Werlberger, and D. Scaramuzza. Svo: Semidirect visual odometry for monocular and multicamera systems. IEEE Transactions on Robotics, 33(2):249-23 265, 2016. [19] Y. Furukawa and C. Hernández. Multi-view stereo: A tutorial. Foundations and Trends@ Computer Graphics and Vision, 9(1-2)11-148, 2015.
[20] C. Kerl, Sturm, and D. Cremers. Dense visual slam for rgb-d cameras. In 2013 IEEE/RSJ International
Conference on Intelligent Robots and Systems, pages 2100-2106. IEEE, 2013.
[21] A. Knapitsch. J. Park. Q.-Y. Zhou, and V. Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics (ToG), 36(4):1-13, 2017.
[22] J. Kopf, X. Rong, and J.-B. Huang. Robust consistent video depth estimation. arXiv preprint arXiv:2012.05900 2020.
[23] J. Krishna Murthy, S. Saryazdi, G. Iyer, and L. Paull. gradslam: Dense slam meets automatic differentiation.
arXiv, 2020.
[24] C. Liu,, J. Gu, K. Kim, S. G. Narasimhan, and J. Kautz, Neural rgh (r) d sensing: Depth and uncertainty from video camera. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 10986-10995, 2019.
[25] X. Luo, J. Huang, R. Szeliski, K. Matzen, and J. Kopf. Consistent video depth estimation. ACM Transactions Graphics (TOG), 39(4) 2020.
10

---

[26] Z. Luo. T. Shen, L. Zhou, S. Zhu, R. Zhang, Y. Yao. T. Fang, and L. Quan. Geodesc: Learning local descriptors by integrating geometry constraints. In Proceedings the European conference on computer vision (ECCV). pages 168-183, 2018.
[27] H. Matsuki, L. von Stumberg, V. Usenko, J. Stückler. and D. Cremers. Onnidirrctional dso: Direct sparse
odometry with fisheye cameras. IEEE Robotics and Automation Letters, 3(4):3693-3700, 2018.
[28] Z. Min, Y. Yang, and E. Dunn. Voldor: Visual odometry from log-logistic dense optical flow residuals. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages -999, 2020.
[29] A. Mishchuk, D. Mishkin, F. Radenovic, and J. Matas. Working hard to know your neighbor's margins: Local descriptor learning loss. arXiv preprint 1705 2017.
[30] A. I. Mourikis and S. I. Roumeliotis. A multi-state constraint kalman filter for vision-aided inertial navigation. In Proceedings 2007 IEEE International Conference on Robotics and Automation, pages
3565-3572. IEEE, 2007.
[31] R. Mur-Artal, J. M. M. Montiel, and J. D. Tardos. Orb-slam: a versatile and accurate monocular slam system. IEEE trunsactions on robotics,
2015.
[32] R. Mur-Artal and D. Tardós. Orb-slam2: An open-source slam system for monocular, stereo, and rgh-d cameras. IEEE Transactions on Robotics, (25)::2255-222,,200
[33] P. K. Nathan Silberman, Derek Hoiem and R. Fergus. Indoor segmentation and support inference from rghd images. In F.CCV, 2012.
[34] R. A. Newcombe, S. J. Lovegrove, and A. Davison. Dtam: Dense tracking and mapping in real-time. In
2011 international conference on computer vision, pages 2320-2327 IEEE, 2011.
[35] Y. Ono. E. Trulls, P. Fua, and K. M. Yi. Lf-net: Learning local features from images. arXiv preprint arKiv: 180.06660 2018.
[36] R. Ranfil and V. Koltun. Deep fundamental matrix estimation. In Proceedings of the European conference computer vision (ECCV), pages 284-299, 2018.
[37] A. Rosinol, M. Abate, Y. Chang, and L. Carlone. Kimera: an open-souree library for real-time metric- semantic localization and mapping. In 2020 IEEE International Conference on Robotics and Automation
(ICRA). pages 1689-1696. IEEE, 2020.
[38] P.-E. Sarlin, D. DeTone, T. Malisiewiez, and A. Rabinovich. Superglue: Learning feature matching with graph neural networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4938-4947, 2020.
[39] P.-E. Sarlin, A. Unagar, M. Larsson, H. Germain, C. Toft, V. Larsson, M. Pollefeys, V. Lepetit, L. Ham- marstrand, F. Kahl, et al. Back to the feature: Learning robust camera localization from pixels to pose. arXiv preprint arXiv::1033 103.09213. 2022.
2021.
[40] J. L. Schonberger and J.-M. Frahm. Structure-frrot from-motion revisited, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4104-411 2016.
[41] T. Schops, T. Sautle,, and M. Pollefeys. Bad slam: Bundle adjusted direct rgb-d slam. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 134-144, 2019.
[42] D. Schubert, N. Demmel, V. Usenko, Stuckler, and D. Cremers. Direct sparse odometry with rolling shutter. Proceedings of the European Conference on Computer Vision (ECCV), pages 682-697, 2018. [43] J. Sturm, N. Engelhard, F. Endres, W. Burgard, and D. Cremers. A benchmark for the evaluation of rgb-d slam systems. 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, pages 973-580.11 IEEE, 2012.
[44] E. Sucar, S. Liu, J. Ortiz, and A. J. Davison. imap: Implicit mapping and positioning in real-time. arXiv preprint Xrx::22103. 12352, 2021.
[45] E. Sucar, K. Wada, and A. Davison. Nodeslam: Neural object descriptors for multi-view shape reconstruc-
tion. In 2020 International Conference on 3D Vision (3DV), pages 949-958. IEEE, 2020.
[46] C. Tang and P. Tan. Ba-net: Dense bundle adjustment network. arXiv preprint arXiv: 1806004877, 2018.
[47] Z. Teed and J. Deng. Deepy2d: Video to depth with differentiable structure from motion. arXiv preprint
Xiv: 181: 8.046605 2018.
[48] Z. Teed and J. Deng. Raft: Recurrent all-pairs field transforms for optical flow. In European Conference
on Computer Vision, pages 402-419. Springer, 2020.
[49] 7. Teed and J. Deng. Tangent space backpropagation for 3d transformation groups. In Conference on Computer Vision and Pattern Recognition, 2021.
[50] B. Ummenhofer, H. Zhou, J. Uhrig, N. Mayer, F. Tlg, A. Dosovitskiy, and T. Brox. Demon: Depth and motion network for learning monocular stereo. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5038-5047, 2017.
[5]] L. von Stumberg, P. Wenzel, N. Yang, and D. Cremers. Lm-reloc: Levenberg marquardt based direct visual relocalization. arXiv preprint 2xXiv:20 2020.
[52] S. Wang, R. Clark, H. Wen, and N. Trigoni. Deepvo: Towards end-to-end visual odometry with deep recur-
rent convolutional neural networks. In 2017 IEEE International Conference on Robotics and Automation (ICRA). pages 2043-2050. IEEE, 2017.

---

[53] W. Wang. Y. Hu, and S. Scherer. Tartanvo: A generalizable learning-based vo. arXiv preprint arXiv:2011.00333 2020.
[54] W. Wang, D. Zhu, X. Wang. Y. Hu, Y. Qiu, C. Wang. Y. Hu, A. Kapoor, and S. Scherer. Tartanair:
dataset to push the limits of visual slam. arXiv preprint arXiv:2003. 14000003333. 2020.
[55] T. Whelan, M. Kaess, Johannsson, Fallon, J. Leonard. J. McDonald. Real-time large-scale dense rgb-d slam with volumetric fusion. The International Journal of Robotics Research, 34(4-5):598-66 626,
2015.
[56] T. Whelan, S. Leutenegger, Salas-Moreno, Glocker, A. Davison. Elasticfusion: Dense slam without a pose graph. Robotics: Science and Systems, 2015.
[57] N. Yang. L. Stumberg. R. Wang, and D. Cremers. D3vo: Deep depth, deep pose and deep uncertainty monocular visual odometry. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 128 1-192, 2020.
[58] N. Yang. Wang. J. Stuckler, and D. Cremers. Deep virtual stereo odometry: Leveraging deep depth prediction monocular direct sparse odometry. Proceedings of European Conference on Computer Vision (ECCV), pages 817-833, 2018.
[59] H. Zhou, B. Ummenhofer. and T. Brox. Deeptam: Deep tracking and mapping. In Proceedings of the Earopean conference on computer vision (ECCV), pages 822-838, 2018.
[60] J. Zubizarreta, I. Aguinaga, and J. M. M. Montiel. Direct sparse mapping. IEEE Transuctions on Robotics.
36(4):1363-13700 2020.
12


Reviews:
[{"review_id": "h2yvjEUTNos", "reviewer": "Reviewer_GZge", "summary": "This paper presents R-SLAM, a learning-based SLAM method that shows state-of-the-art camera pose estimation performance on several benchmarks. In particular, R-SLAM updates the camera poses and dense frame depth in an iterative manner by using a recurrent update operator built upon RAFT and differentiable bundle adjustment. To make a complete SLAM system, R-SLAM imposes a covisibility graph and conducts both local and global optimization. The experiments show that R-SLAM outperforms the previous classical and learning-based approaches by a large margin, even on the datasets on which it is not trained.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strength\n1. The proposed SLAM method, R-SLAM, has shown the state-of-the-art camera pose estimation results on several established benchmarks, including TantanAir, EuRoC, TUM RGB-D, and ETH3D. Also, R-SLAM shows in general quite large improvements over the prior works, e.g., by ~60% on TantanAir, ~40% on EuRoC, ~80% on TUM RGB-D, and ~35% on ETH3D.\n2. The state-of-the-art results demonstrated are obtained with the model trained on TantanAir only, thus showing very strong generalization capability which is often the major limitation of learning-based SLAM approaches.\n3. R-SLAM takes advantage of both classical optimization and deep learning -- it can be trained end-to-end while keeping the bundle adjustment (BA) with Gauss-Newton in the loop. Besides, R-SLAM conducts the joint optimization of pose and depth which has been shown to be one of the keys to accurate estimations.\n4. Besides the high accuracy, R-SLAM also shows improved robustness with a high success rate on several datasets.\n\n## Weaknesses\n1. In general, the novelty of the paper is not substantial. It builds upon the established systems and methodologies -- a hybrid SLAM design that combines both the classical and learning worlds, differentiable BA / Gauss-Newton layer, covisibility graph, and the recurrent update operator. Overall it feels like a system as DeepV2D + RAFT, and no fundamental theoretical or architectural innovations are proposed. Above being said, I believe combining the previous works and making the right choices that lead to notable improvements is still quite valuable and inspiring to the community.\n2. R-SLAM is trained on TatanAir while most of the compared methods (DeepFactors, DeepV2D, D3VO, DeepTAM) are not. Although some of the aforementioned methods are trained with the train/test split on the same data while R-SLAM shows mostly the cross-dataset generalization, it still poses a certain level of unfairness on the comparisons since, in general, the TantanAir dataset contains more images and more diversities. As a suggestion, it would be better to show a method trained and evaluated using exactly the same dataset as R-SLAM and report the comparison, for which DeepV2D could be a good baseline.\n3. It is unclear how the non-keyframe poses are estimated if they are estimated at all. Since it is a SLAM system instead of an SfM system, R-SLAM should be able to deliver both keyframe and non-keyframe poses for real-world applications. However, from the paper one cannot see how the non-keyframe poses are handled. In fact, this infers other issues on the evaluation -- R-SLAM is evaluated with the keyframe pose without non-keyframe poses, but is it also the case for the other compared methods?; are the poses used in the evaluations the outputs of the global BA? If so, how good are the poses from local bundle adjustment since some compared methods are Visual Odometry systems without any global BA, e.g, DeepV2D, D3VO, DSO, etc?\n4. The details of the full dense bundle adjustment are missing. R-SLAM performs joint dense BA on pose and depth which is quite expensive, especially when doing the global BA which optimizes tens or even hundreds of frame depth and poses. Is there a customized GPU-based optimizer used?\n5. It would be better if more qualitative results can be shown, e.g., camera trajectories comparison among the methods and the estimated dense depth maps.\n6. The writing of the paper can still be improved. There is a number of typos in the current manuscript. I list some of them here but there should be more: \n- L10: \"focused\" -> \"focuses\"\n- L13: \"approach\" -> \"approached\"\n- L15: \"are have\" -> \"have\"\n- L20: \"But despite\" -> \"Despite\" -\n- L57: \"solves computes\" -> \"computes\" \n- L81: \"is that features\" -> remove \n- L115: \"operators\" -> \"operates\"\n- Eq. (5): $\\triangle w$ -> $\\triangle d$?\n7. The name \"R-SLAM\" already exists in [1] and [2]. In order to avoid confusion in the future, I would suggest the authors proposing a new name of the system.\n\n## Overall Comment\nI recommend accepting the paper due to the impressive results and the reasonable design choices made by the authors. But I still would like to see the authors' responses to the points listed in the Weaknesses section above.\n\n----------\nReferences:\n1. Balcılar, Muhammet, et al. \"R-SLAM: Resilient localization and mapping in challenging environments.\" Robotics and Autonomous Systems 87 (2017): 66-80.\n2. Mei, Christopher, et al. “RSLAM: A system for large-scale mapping in constant-time using stereo. International journal of computer vision 94 (2), 198-214, 2011", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors have addressed memory consumption as one of the limitations of the method.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "GqpiXL8Eqw4", "reviewer": "Reviewer_vRNK", "summary": "This paper proposes a novel deep learning-based SLAM algorithm. The key technical contribution is to adopt a recurrently updated deep optical flow to serve as the estimation of correspondence and build a differentiable bundle adjustment solver module within the network. The proposed approach is solid and technically sound. It sets the new state-of-the-art and demonstrates exciting potential opportunities. The paper is clearly presented. I would strongly recommend accepting this paper. \n", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n* Very strong results. Significantly better than prior art on TartanAir, EuRoC, ETH3D, and TUM and close to being real-time. It's exciting to see such results on SLAM. \n* The approach is simple, elegant, and makes a lot of sense. I am particularly interested in the iterative learning of the residual flow. \n* Less hyper-parameter tuning and engineering trick than most of the existing visual SLAM approaches. \n* Easy to generalize to monocular depth, stereo, and RGB-d settings. It also demonstrates generalization capability across different datasets. \n\n\nCons/ Detailed Comments:\n\n\nWriting:\n* I would not claim the DBA layer is new — as you pointed out in the paper and some other existing literature, very similar instantiation has been seen before. It won’t hurt even if this part is not entirely novel — this submission has a good and novel design of the entire SLAM approach. \n* There are not enough discussions to justify the technical choice and superior performance: 1) why rejection error is superior to the photometric ones as in indirect method; 2) why dense optical flow than sparse key feature correspondence? 3) why do you need the residual updates of the flow estimation? \n* Do you see any failure cases in some testing videos? Could you show some failure cases?\n* Qualitative results are not sufficient for this paper. I would like to see visualizations of the flow confidence map, intermediate flow & depth results, iterative updates, before vs after local BA, and an online demo video in the final supplementary. \n* \n\nApproaches:\n* It’s not clear to me how did you select pixels for the local bundle adjustment (all or prune some by the confidence weight?) and how dense you store the features for global BA. \n* Many papers have seen superior results using differentiable LM solver than GN solver, with a proper damping factor scheme. Have you compared both? \n* Do you need a good initial pose estimation for a new incoming frame at the inference?\n* Initialization from the first 12 frames is interesting to me: how do you initialize your pose then? All zeros? Won’t they fall into some local optimal? If not, do you have some intuitions why? This is different from the conventional approach, where we might need a good init from homography or essential decomposition. \n* Have you double-checked whether Schur complement is indeed superior to other options, e.g., Cholesky in this case? Given the structure, Cholesky could be faster as well. \n* It seems you directly remove the keyframe, have you considered conducting marginalization using the scour complement? I would love to see whether it makes a difference here. \n* Is there any way for the network to decide on a tracking failure in some cases? This will be very useful in practice. Is the learned energy value a good indicator of success? \n\n\nExperiments:\n* There isn’t much ablation on the tech choice, e.g., optical flow-based results vs. only using dense depth-based results; different options for local BA windows. \n* Following the above discussion on flow vs. feature matching, I am actually wondering how well superpoint+superglue + your proposed frontend DBA layer + your backend do. This would be a nice experiment. \n* The scale of the global bundle adjustment is still very big. How do you deal with it? Do you directly implement with PyTorch tensors? \n* Given the sparse block patterns, have you considered using a sparse tensor in the implementation to reduce memory footage and accelerate, like what has been done in ceres/gtsam? \n* Is it straightforward to integrate a loop-closure module in the backend? \n\n\nRelevant works:\n* gradSLAM: Automagically differentiable SLAM, CVPR 2020: this is a similar end-to-end approach based on a differentiable bundle adjustment solver. That being said, both frontend and backend are different, and this submission’s result is stronger. \n* Deep rigid instance scene flow, CVPR 2019; and RAFT-3D in CVPR 2020: these approaches have a similar DBA-layer optimizing reprojection error. The correspondence is based on deep flow estimation, but this approach is two-frame and mainly focuses on instance-level 3D scene flow. Besides, it would help if you discussed VOLSOR and Flowfusion more explicitly in related work, as they are also deep optical flow-based slam and directly relevant. This helps better position the paper and acknowledge prior work.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"ethical_concerns": "NA", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "D9b7gTzdlzg", "reviewer": "Reviewer_2zGu", "summary": "This paper learns to estimate the camera poses and depth by interlacing the residual flow field estimation and the geometric bundle adjustment iteratively. At each iteration, a residual flow field is estimated pair-wisely between two frames, and then a bundle adjustment step is applied on all residual flow fields anchored at a keyframe to update the current camera poses and depth. The experiments demonstrate the superiority of the proposed method over existing learning based methods and conventional methods on various datasets.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n1. Interlacing between 2D residual flow and 3D camera poses and depth is a nice design to tackle the SLAM problem.\n2. The experiments are comprehensive and adequate. The results on various datasets demonstrate the generalization ability of the residual flow estimation.\n\nWeakness:\n\nIn general, there is no critical weakness except for some issues that need to be further clarified:\n1. The paper does not mention the training data source. It describes how the data is constructed in sec.3.3, but what are the datasets that are used to produce the training data?  Are they from the same dataset as evaluation or they are completely different such as a synthetic dataset? This issue matters to decide how significant the generalization is.\n2. The method mentioned that there is a weight predicted in line-177. This weight prediction is important because the method interlacing between residual flow field and bundle adjustment, which is sensitive to outliers such as the ones caused by dynamic objects. However, there is no visualization or evaluation about this weighting in the paper. This issue also depends on the training data that if the training data are all static scenes how it can weigh properly on sequences with dynamic objects? So it would be good to have a clear discussion and analysis in the paper.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The computation limitation has been discussed in line-322. However, the limitation about outlier handling is not mentioned. \nConventionally, we can do outlier rejection after correspondences have been estimated and then recover camera poses and depth from bundle adjustment. However, in this paper, the flow estimation and bundle adjustment are interlaced. At each iteration, some outlier points such as dynamic objects will always be projected incorrectly and further affect residual flow estimation and bundle adjustment. Unless the weight prediction has been demonstrated to be sufficiently powerful, otherwise it will also be a limitation for this method. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3.5"}, "extra_text": {}}, {"review_id": "8XhS2VuUGh", "reviewer": "Reviewer_YJaQ", "summary": "This paper does dense Structure from Motion using a data-driven dense Bundle Adjustment layer, that is optimized using flow fields. Experiments conducted show that this method outperforms the state of the art and is generalizable to new datasets (which is rare for a learning based method).", "questions": "", "limitations": "", "rating": 9, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper builds on the work done in the RAFT [46] paper, that generates dense optic-flow-fields using an iterative optimization procedure. A 4D dense, multi-scale correlation volume (that is a representation of regions of similarity) is built on pairs of images and the flow-field (and its associated uncertainty) between the pair of images is built up iteratively from this, using a recurrent (GRU) unit. This is like a data-driven, learnt optimization process, that iteratively approaches the correct optic flow given lots of video sequences. However, unlike the RAFT paper, this is not learnt using ground truth flow. In this work, because the structure and motion in the scene needs to be estimated, this is trained using ground truth depth and poses [this is not entirely clear and should be explained better that the ground truth depth and pose are required during training - if I understand it right]. The iterative updates to the optic flow from the GRU unit is in turn used by a Dense Bundle Adjustment (DBA) layer to iteratively arrive at a dense depth map and pose estimate for a pair of images. DBA uses the cost function over a sequence of frames, with flow error (between what is predicted from the GRU and what is obtained from the current estimates of depth and pose) that is optimized using a Gauss-Newton algorithm. It is an important distinction from prior work in this area like BA-net, that used photometric error for optimization and this work uses the flow error. This presumably leads to better robustness to lighting changes and camera blur, etc. The entire network is trained end-to-end using a combination of pose loss and flow loss. \n\nExperiments conducted indicate the ability of the system to surpass SoTa. A network trained on the Tartan Air dataset generalizes to the EuRoC and TUM-RGBD datasets, and improves on the ATE on those datasets by as much as 83%. \nThis is also achieved in real-time or near-real-time run-time on desktop hardware.\n\nThis paper is well written and is an impressive result and a significant contribution to the field. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Some minor grammatical errors that need correction:\n126: they\n143: show\n176: 'convoluation'\n227: an\n231: 'activly begin'\n\nAround line 123, it is mentioned that the frame graph is built using frame co-visibility, but it is not immediately clear as to how this co-visibility is calculated. It is mentioned later in the paper (207) that this is done by thresholding optic flow. This also needs to be mentioned here (around line 123). It is also not very clear how optic flow is used while training - when presumably the network's own optic flow capabilities will be non-existent (at the beginning of training). Is a pre-trained optic flow network being used for this?\n\nSupervision (209): mentions that a combination of pose loss and flow loss is used and that ground truth poses and depth is used. Does this mean that this network cannot be trained using monocular sequences? Or would you use self-supervised mono-depth techniques to get the depth?\n\n", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 "}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received strong reviews and I am happy to recommend it for acceptance. I encourage the authors to make the improvements that they indicated in their rebuttal in the final version. 
Decision: Accept (Oral)