DeepSFM: Structure From Motion Via Deep Bundle Adjustment

1 INTRODUCTION . Structure from motion ( SfM ) is a fundamental human vision functionality which recovers 3D structures from the projected retinal images of moving objects or scenes . It enables machines to sense and understand with the 3D world and is critical in achieving real-world artificial intelligence . Over decades of researches , there has been a lot of great success on SfM ; however , the performance is far from perfect . Conventional SfM approaches ( Agarwal et al. , 2011 ; Wu et al. , 2011a ; Engel et al. , 2017 ; Delaunoy & Pollefeys , 2014 ) heavily rely on Bundle-Adjustment ( BA ) ( Triggs et al. , 1999 ; Agarwal et al. , 2010 ) , in which 3D structures and camera motions of each view are jointly optimized via LevenbergMarquardt ( LM ) algorithm ( Nocedal & Wright , 2006 ) according to the cross-view correspondence . Though successful in certain scenarios , conventional SfM based approaches are fundamentally restricted by the coverage of the provided multiple views and the overlaps among them . They also typically fail to reconstruct textureless or non-lambertian ( e.g . reflective or transparent ) surfaces due to the missing of correspondence across views . As a result , selecting sufficiently good input views and the right scene requires excessive caution and is usually non-trivial to even experienced user . Recent researches resort to deep learning to deal with the typical weakness of conventional SfM . Early effort utilizes deep neural network as a powerful mapping function that directly regresses the structures and motions ( Ummenhofer et al. , 2017 ; Vijayanarasimhan et al. , 2017 ; Zhou et al. , 2017 ; Wang et al. , 2017 ) . Since the geometric constraints of structures and motions are not explicitly enforced , the network does not learn the underlying physics and prone to overfitting . Consequently , they do not perform as accurate as conventional SfM approaches and suffer from extremely poor generalization capability . Most recently , the 3D cost volume ( Teed & Deng , 2018 ) has been introduced to explicit leveraging photo-consistency in a differentiable way , which significantly boosts the performance of deep learning based 3D reconstruction . However , the camera motion usually has to be known ( Yao et al. , 2018 ; Im et al. , 2019 ) or predicted via direct regression ( Ummenhofer et al. , 2017 ; Zhou et al. , 2017 ; Teed & Deng , 2018 ) , which still suffer from generalization issue . In this paper , we explicitly enforce photo-consistency , geometric-consistency , and camera motion constraints in a unified deep learning framework . In particular , our network includes a depth based cost volume ( D-CV ) and a pose based cost volume ( P-CV ) . D-CV optimizes per-pixel depth values with the current camera poses , while P-CV optimizes camera poses with the current depth estimations . Conventional 3D cost volume enforces photo-consistency by unprojecting pixels into the discrete camera fronto-parallel planes and computing the photometric ( i.e . image feature ) difference as the cost . In addition to that , our D-CV further enforces geometric-consistency among cameras with their current depth estimations by adding the geometric ( i.e . depth ) difference to the cost . Note that the initial depth estimation can be obtained using the conventional 3D cost volume . For pose estimation , rather than direct regression , our P-CV discretizes around the current camera positions , and also computes the photometric and/or geometric differences by hypothetically moving the camera into the discretized position . Note that the initial camera pose can be obtained by a rough estimation from the direct regression methods such as ( Ummenhofer et al. , 2017 ) . Our framework bridges the gap between the conventional and deep learning based SfM by incorporating explicit constraints of photo-consistency , geometric-consistency and camera motions all in the deep network . The closest work in the literature is the recently proposed BA-Net ( Tang & Tan , 2018 ) , which also aims to explicitly incorporate multi-view geometric constraints in a deep learning framework . They achieve this goal by integrating the LM optimization into the network . However , the LM iterations are unrolled with few iterations due to the memory and computational inefficiency , and thus it may lead to non-optimal solutions . In contrast , our method does not have a restriction on the number of iterations and achieves empirically better performance . Furthermore , LM in SfM originally optimizes point and camera positions , and thus direct integration of LM still requires good correspondences . To evade the correspondence issue in typical SfM , their models employ a direct regressor to predict depth at the front end , which heavily relies on prior in the training data . In contrast , our model is a fully physical-driven architecture that less suffers from over-fitting issue for both depth and pose estimation . To demonstrate the superiority of our method , we conduct extensive experiments on DeMoN datasets , ScanNet and ETH3D . The experiments show that our approach outperforms the state-ofthe-art Schonberger & Frahm ( 2016 ) ; Ummenhofer et al . ( 2017 ) ; Tang & Tan ( 2018 ) . 2 RELATED WORK . There is a large body of work that focuses on inferring depth or motion from color images , ranging from single view , multiple views and monocular video . We discuss them in the context of our work . Single-view Depth Estimation . While ill-posed , the emerging of deep learning technology enables the estimation of depth from a single color image . The early work directly formulates this into a per-pixel regression problem ( Eigen et al. , 2014 ) , and follow-up works improve the performance by introducing multi-scale network architectures ( Eigen et al. , 2014 ; Eigen & Fergus , 2015 ) , skipconnections ( Wang et al. , 2015 ; Liu et al. , 2016 ) , powerful decoder and post process ( Garg et al. , 2016 ; Laina et al. , 2016 ; Kuznietsov et al. , 2017 ; Wang et al. , 2015 ; Liu et al. , 2016 ) , and new loss functions ( Fu et al. , 2018 ) . Even though single view based methods generate plausible results , the models usually resort heavily to the prior in the training data and suffer from generalization capability . Nevertheless , these methods still act as an important component in some multi-view systems ( Tang & Tan , 2018 ) Traditional Structure-from-Motion Simultaneously estimating 3d structure and camera motion is a well studied problem which has a traditional tool-chain of techniques ( Furukawa et al. , 2010 ; Newcombe et al. , 2011 ; Wu et al. , 2011b ) . Structure from Motion ( SfM ) has made great progress in many aspects . Lowe ( 2004 ) ; Han et al . ( 2015 ) aim at improving features and Snavely ( 2011 ) introduce new optimization techniques . More robust structures and data representations are introduced by Gherardi et al . ( 2010 ) ; Schonberger & Frahm ( 2016 ) . Simultaneous Localization and Sapping ( SLAM ) systems track the motion of the camera and build 3D structure from video sequence ( Newcombe et al. , 2011 ; Engel et al. , 2014 ; Mur-Artal et al. , 2015 ; Mur-Artal & Tardós , 2017 ) . Engel et al . ( 2014 ) propose the photometric bundle adjustment algorithm to directly minimize the photometric error of aligned pixels . However , traditional SfM and SLAM methods are sensitive to low texture region , occlusions , moving objects and lighting changes , which limit the performance and stability . Deep Learning for Structure-from-Motion Deep neural networks have shown great success in stereo matching and Structure-from-Motion problems . Ummenhofer et al . ( 2017 ) ; Wang et al . ( 2017 ) ; Vijayanarasimhan et al . ( 2017 ) ; Zhou et al . ( 2017 ) regress depth map and camera pose di- rectly in a supervised manner or by introducing photometric constraints between depth and motion as a self-supervision signal . Such methods solve the camera motion as a regression problem , and the relation between camera motion and depth prediction is neglected . Recently , some methods exploit multi-view photometric or feature-metric constraints to enforce the relationship between dense depth map and the camera pose in network . The SE3 transformer layer is introduced by Teed & Deng ( 2018 ) , which uses geometry to map flow and depth into a camera pose update . Wang et al . ( 2018 ) propose the differentiable camera motion estimator based on the Direct Visual Odometry ( Steinbrücker et al. , 2011 ) . Clark et al . ( 2018 ) using a LSTM-RNN ( Hochreiter et al. , 2001 ) as the optimizer to solve nonlinear least squares in two-view SfM . Tang & Tan ( 2018 ) train a network to generate a set of basis depth maps and optimize depth and camera poses in a BA-layer by minimizing a feature-metric error . 3 ARCHITECTURE . Our framework receives frames of a scene from different viewpoints , and produces photo-metrically and geometrically consistent depth maps across all frames and the corresponding camera poses . Similar to BA , we also assume initial structures ( i.e depth maps ) and motions ( i.e . camera poses ) are given . Note that the initialization is not necessary to be super accurate for the good performance using our framework and thus can be easily obtained from some direct regression methods ( Ummenhofer et al. , 2017 ) . Now we introduce the detail of our model – DeepSFM . Without loss of generality , we describe our model taking two images as input , namely the reference image and the source image , as an example , and all the technical components can be extended for multiple images straightforward . As shown in Figure 1 , we first extract feature maps from input images through a shared encoder . We then sample the solution space for depth uniformly in the inverse-depth space between a predefined minimum and maximum range and camera pose around the initialization respectively . After that , we build cost volumes accordingly to reason the confidence of each hypothesis . This is achieved by validating the consistency between the feature of the reference view and the ones warped from the source image . Besides photo-metric consistency that measures the color image similarity , we also take into account the geometric consistency across warped depth maps . Note that depth and pose require different designs of cost volume to efficiently sample the hypothesis space . Gradients can back-propagate through cost volumes , and cost-volume construction does not affect any trainable parameters . The cost volumes are then fed into 3D CNN to regress new depth and pose . These updated value can be used to create new cost volumes , and the model improves the prediction iteratively . For notations , we denote { Ii } ni=1 as the image sequences in one scene , { Di } ni=1 as the corresponding ground truth depth maps , { Ki } ni=1 as the camera intrinsics , { Ri , ti } ni=1 as the ground truth rotations and translations of camera , { D∗i } ni=1 and { R ∗ i , t∗i } ni=1 as initial depth maps and camera pose parameters for constructing cost volumes , where n is the number of image samples . 3.1 2D FEATURE EXTRACTION . Given the input sequences { Ii } ni=1 , we extract the 2D CNN feature { Fi } ni=1 for each frame . Firstly , a 7 layers ’ CNN with kernel size 3×3 is applied to extract low contextual information . Then we adopt a spatial pyramid pooling ( SPP ) ( Kaiming et al. , 2014 ) module , which can extract hierarchical multiscale features through 4 average pooling blocks with different pooling kernel size ( 4×4 , 8×8 , 16× 16 , 32 × 32 ) . Finally , we pass the concatenated features through 2D CNNs to get the 32-channel image features after upsampling these multi-scale features into the same resolution . These image sequence features are used by the building of both our depth based and pose based cost volumes .

Summary: The paper tackles Structure from Motion, one of the canonical problems in computer vision, and proposes an approach that brings together geometry and physics on one hand and deep networks on the other hand. Camera unprojection and warping (of depth maps and features) are used to build a cost volume onto hypothetical planes perpendicular to the camera axis. Similarly, various camera poses are sampled around an initial guess. A deep network regresses form the cost volume to a camera pose and a depth map. The method can be applied iteratively, using the outputs of the current stage as the initial guess of the next one. Training is supervised, and the the results are evaluated on multiple datasets.