DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras

1 Introduction . Simultaneous Localization and Mapping ( SLAM ) aims to ( 1 ) build a map of the environment and ( 2 ) localize the agent within the environment . It is a special form of Structure-from-Motion ( SfM ) focused on accurate tracking of long-term trajectories . It is a critical capability for robotics , especially autonomous vehicles . In this work , we address visual SLAM , where sensor recordings come in the form of images captured from a monocular , stereo , or RGB-D camera . The SLAM problem has been approached from a number of different angles . Early work was built using probabilistic and filtering based approaches [ 12 , 30 ] , and alternating optimization of the map and camera poses [ 34 , 16 ] . More recently , modern SLAM systems have leveraged least-squares optimization . A key element for accuracy has been full Bundle Adjustment ( BA ) , which jointly optimizes the camera poses and the 3D map in a single optimization problem . One advantage of the optimization-based formulation is that a SLAM system can be easily modified to leverage different sensors . For example , ORB-SLAM3 [ 5 ] supports monocular , stereo , RGB-D , and IMU sensors , and modern systems can support a variety of camera models [ 5 , 27 , 42 , 6 ] . Despite significant progress , current SLAM systems lack the robustness demanded for many real-world applications . Failures come in many forms , such as lost feature tracks , divergence in the optimization algorithm , and accumulation of drift . Deep learning has been proposed as a solution to many of these failure cases . Previous work has investigated replacing hand-crafted with learned features [ 13 , 7 , 29 , 26 , 35 ] , using neural 3D representations [ 46 , 1 , 9 , 45 , 44 , 25 , 22 ] , and combining learned energy terms with classical optimization backends [ 58 , 57 ] . Other work has tried to learn SLAM or VO systems end-to-end [ 59 , 47 , 53 , 52 , 46 ] . While these systems are sometimes more robust , they fall far short of the accuracy of their classical counterparts on common benchmarks . In this work we introduce DROID-SLAM , a new SLAM system based on deep learning . It has state-of-the-art performance , outperforming existing SLAM systems , classical or learning-based , on challenging benchmarks with very large margins . In particular , it has the following advantages : • High Accuracy : We achieve large improvements over prior work across multiple datasets and modalities . On the TartanAir SLAM competition [ 54 ] , we reduce error by 62 % over the best prior result on the monocular track and 60 % on the stereo track . We rank 1st on the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . ETH-3D RGB-D SLAM leaderboard [ 41 ] , outperforming the second place by 35 % under the AUC metric which considers both error and rate of catastrophic failure . On EuRoC [ 2 ] , with monocular input , we reduce error by 82 % among methods with zero failures , and by 43 % over ORB-SLAM3 considering only the 10 out of 11 sequences it succeeds on . With stereo input , we reduce error by 71 % over ORB-SLAM3 . On TUM-RGBD [ 43 ] , we reduce error by 83 % among the methods with zero failures . • High Robustness : We have substantially fewer catastrophic failures than prior systems . On ETH-3D , we successfully track 30 of the 32 RGB-D datasets , while the next best successfully tracks only 19/32 . On TartanAir , EuRoC , and TUM-RGBD , we have zero failures . • Strong Generalization : Our system , trained only with monocular input , can directly use stereo or RGB-D input to get improved accuracy without any retraining . All of our results across 4 datasets and 3 modalities are achieved by a single model , trained once with only monocular input entirely on the synthetic TartanAir dataset . The strong performance and generalization of DROID-SLAM is made possible by its “ Differentiable Recurrent Optimization-Inspired Design ” ( DROID ) , which is an end-to-end differentiable architecture that combines the strengths of both classical approaches and deep networks . Specifically , it consists of recurrent iterative updates , building upon RAFT [ 48 ] for optical flow but introducing two key innovations . First , unlike RAFT , which iteratively updates optical flow , we iteratively update camera poses and depth . Whereas RAFT operates on two frames , our updates are applied to an arbitrary number of frames , enabling joint global refinement of all camera poses and depth maps , essential for minimizing drift for long trajectories and loop closures . Second , each update of camera poses and depth maps in DROID-SLAM is produced by a differentiable Dense Bundle Adjustment ( DBA ) layer , which computes a Gauss-Newton update to camera poses and dense per-pixel depth so as to maximize their compatibility with the current estimate of optical flow . This DBA layer leverages geometric constraints , improves accuracy and robustness , and enables a monocular system to handle stereo or RGB-D input without retraining . The design of DROID-SLAM is novel . The closest prior deep architectures are DeepV2D [ 47 ] and BA-Net [ 46 ] , both of which were focused on depth estimation and reported limited SLAM results . DeepV2D alternates between updating depth and updating camera poses , instead of bundle adjustment . BA-Net has a bundle adjustment layer , but their layer is substantially different : it is not “ dense ” in that it optimizes over a small number of coefficients used to linearly combine a depth basis ( a set of pre-predicted depth maps ) , whereas we optimize over per-pixel depth directly , without being handicapped by a depth basis . In addition , BA-Net optimizes photometric reprojection error ( in feature space ) , whereas we optimize geometric error , leveraging state-of-the-art flow estimation . We perform extensive evaluation across four different datasets and three different sensor modalities , demonstrating state-of-the-art performance in all cases . We also include ablation studies that shed light on important design decisions and hyperparameters . 2 Related Work . Modern SLAM systems treat localization and mapping as a joint optimization problem [ 4 ] . Visual SLAM focuses on observations in the form of monocular , stereo , or RGB-D images . These approaches are commonly categorized as either being direct or indirect [ 15 ] . Indirect approaches [ 31 , 32 , 5 , 37 ] first process the image into an intermediate representation by detecting points of interest and attaching feature descriptors . Features are then matched between images . Indirect approaches optimize camera pose and a 3D point cloud by minimizing reprojection error–the distance between a projected 3D point and its location in the image . Direct approaches model the image formation process and define an objective function over photometric error [ 16 , 15 , 60 ] . One advantage of direct approaches is that they can model more information about the image , such as lines and intensity variations [ 15 ] which are not used by indirect approaches . However , photometric errors typically lead to more difficult optimization problems , and direct approaches are less robust to geometric distortion such as rolling shutter artifacts . This approach requires more sophisticated optimization techniques , such as coarse-to-fine image pyramids to avoid local minimum . Our method does not clearly fit into either of the categories . Like the direct approach , we do not require preprocessing steps to detect and match features between the images . We instead use the full image , allowing us to leverage a wider range of information than indirect methods with typically only use corners and edges . However , we minimize reprojection error similar to indirect methods . This is an easier optimization problem and avoids the need for more complicated representations such as image pyramids . In this sense , our approach borrows the best of both approaches : the smoother objective function of indirect approaches with the greater modeling capacity of indirect approaches . Deep Learning has more recently been applied to the SLAM problem . Many works have focused on training systems for particular subproblems , such as feature detection [ 13 , 7 , 29 , 26 , 35 ] , feature matching and outlier rejection [ 38 , 36 ] , and localization [ 51 , 39 ] . SuperGlue [ 38 ] was designed to perform feature matching and verification and make 2-view pose estimate much more robust . Our network also draws inspiration from Dusmanu et al [ 14 ] , which builds a neural network into the SfM pipeline to improve keypoint localization accuracy . Other works have focused on training SLAM systems end-to-end [ 59 , 46 , 8 , 50 , 24 , 47 , 53 ] . These methods are not full SLAM systems , but instead focus on small scale reconstruction on the order of two [ 8 , 50 , 53 ] up to a dozen frames [ 59 , 46 , 47 ] . They lack many of the core capabilities of modern SLAM systems such as loop closure and global bundle adjustment which inhibit their ability to perform large scale reconstruction as demonstrated in our experiments . ∇SLAM [ 23 ] implements several existing SLAM algorithms as differentiable computation graphs , allowing for errors in the reconstruction to be backpropagated back to sensor measurements . While this approach is differentiable , it has no trainable parameters , meaning the performance of the system is limited by the accuracy of the classical algorithm they emulate . DeepFactors [ 9 ] is the most complete deep SLAM system , building on the earlier CodeSLAM [ 1 ] . It performs joint optimization of the pose and depth variables , and is capable of short and long-range loop closure . Similar to BA-Net [ 46 ] , DeepFactors optimizes the parameters of a learned depth basis during inference . In contrast , we do not rely on a learned basis , but instead optimize pixelwise depth . This allows our network to better generalize to new datasets since our depth representation is not tied to the training dataset . 3 Approach . We take a video as input with two objectives : estimate the trajectory of the camera and build a 3D map of the environment . We first describe the monocular setting ; in Sec . 3.4 we describe how to generalize the system to stereo and RGB-D video . Representation : Our network operates on an ordered collection of images , { It } Nt=0 . For each image t , we maintain two state variables : camera pose Gt ∈ SE ( 3 ) and inverse depth dt ∈ RH×W+ . The set of poses , { Gt } Nt=0 , and set of inverse depths { dt } Nt=0 are unknown state variables , which get iteratively updated during inference as new frames are processed . For the reminder of the paper , when we refer to depths , note that we are using the inverse depth parameterization . We adopt a frame-graph ( V , E ) to represent co-visibility between frames . An edge ( i , j ) ∈ E means image Ii and Ij have overlapping fields of view which shared points . The frame graph is built dynamically during training and inference . After each pose or depth update , we can recompute visibility to update the frame graph . If the camera returns to a previously mapped region , we add long range connections in the graph to perform loop closure .

Summary: This paper learns to estimate the camera poses and depth by interlacing the residual flow field estimation and the geometric bundle adjustment iteratively. At each iteration, a residual flow field is estimated pair-wisely between two frames, and then a bundle adjustment step is applied on all residual flow fields anchored at a keyframe to update the current camera poses and depth. The experiments demonstrate the superiority of the proposed method over existing learning based methods and conventional methods on various datasets.