DeepSFM: Structure From Motion Via Deep Bundle Adjustment

1 INTRODUCTION . Structure from motion ( SfM ) is a fundamental human vision functionality which recovers 3D structures from the projected retinal images of moving objects or scenes . It enables machines to sense and understand with the 3D world and is critical in achieving real-world artificial intelligence . Over decades of researches , there has been a lot of great success on SfM ; however , the performance is far from perfect . Conventional SfM approaches ( Agarwal et al. , 2011 ; Wu et al. , 2011a ; Engel et al. , 2017 ; Delaunoy & Pollefeys , 2014 ) heavily rely on Bundle-Adjustment ( BA ) ( Triggs et al. , 1999 ; Agarwal et al. , 2010 ) , in which 3D structures and camera motions of each view are jointly optimized via LevenbergMarquardt ( LM ) algorithm ( Nocedal & Wright , 2006 ) according to the cross-view correspondence . Though successful in certain scenarios , conventional SfM based approaches are fundamentally restricted by the coverage of the provided multiple views and the overlaps among them . They also typically fail to reconstruct textureless or non-lambertian ( e.g . reflective or transparent ) surfaces due to the missing of correspondence across views . As a result , selecting sufficiently good input views and the right scene requires excessive caution and is usually non-trivial to even experienced user . Recent researches resort to deep learning to deal with the typical weakness of conventional SfM . Early effort utilizes deep neural network as a powerful mapping function that directly regresses the structures and motions ( Ummenhofer et al. , 2017 ; Vijayanarasimhan et al. , 2017 ; Zhou et al. , 2017 ; Wang et al. , 2017 ) . Since the geometric constraints of structures and motions are not explicitly enforced , the network does not learn the underlying physics and prone to overfitting . Consequently , they do not perform as accurate as conventional SfM approaches and suffer from extremely poor generalization capability . Most recently , the 3D cost volume ( Teed & Deng , 2018 ) has been introduced to explicit leveraging photo-consistency in a differentiable way , which significantly boosts the performance of deep learning based 3D reconstruction . However , the camera motion usually has to be known ( Yao et al. , 2018 ; Im et al. , 2019 ) or predicted via direct regression ( Ummenhofer et al. , 2017 ; Zhou et al. , 2017 ; Teed & Deng , 2018 ) , which still suffer from generalization issue . In this paper , we explicitly enforce photo-consistency , geometric-consistency , and camera motion constraints in a unified deep learning framework . In particular , our network includes a depth based cost volume ( D-CV ) and a pose based cost volume ( P-CV ) . D-CV optimizes per-pixel depth values with the current camera poses , while P-CV optimizes camera poses with the current depth estimations . Conventional 3D cost volume enforces photo-consistency by unprojecting pixels into the discrete camera fronto-parallel planes and computing the photometric ( i.e . image feature ) difference as the cost . In addition to that , our D-CV further enforces geometric-consistency among cameras with their current depth estimations by adding the geometric ( i.e . depth ) difference to the cost . Note that the initial depth estimation can be obtained using the conventional 3D cost volume . For pose estimation , rather than direct regression , our P-CV discretizes around the current camera positions , and also computes the photometric and/or geometric differences by hypothetically moving the camera into the discretized position . Note that the initial camera pose can be obtained by a rough estimation from the direct regression methods such as ( Ummenhofer et al. , 2017 ) . Our framework bridges the gap between the conventional and deep learning based SfM by incorporating explicit constraints of photo-consistency , geometric-consistency and camera motions all in the deep network . The closest work in the literature is the recently proposed BA-Net ( Tang & Tan , 2018 ) , which also aims to explicitly incorporate multi-view geometric constraints in a deep learning framework . They achieve this goal by integrating the LM optimization into the network . However , the LM iterations are unrolled with few iterations due to the memory and computational inefficiency , and thus it may lead to non-optimal solutions . In contrast , our method does not have a restriction on the number of iterations and achieves empirically better performance . Furthermore , LM in SfM originally optimizes point and camera positions , and thus direct integration of LM still requires good correspondences . To evade the correspondence issue in typical SfM , their models employ a direct regressor to predict depth at the front end , which heavily relies on prior in the training data . In contrast , our model is a fully physical-driven architecture that less suffers from over-fitting issue for both depth and pose estimation . To demonstrate the superiority of our method , we conduct extensive experiments on DeMoN datasets , ScanNet and ETH3D . The experiments show that our approach outperforms the state-ofthe-art Schonberger & Frahm ( 2016 ) ; Ummenhofer et al . ( 2017 ) ; Tang & Tan ( 2018 ) . 2 RELATED WORK . There is a large body of work that focuses on inferring depth or motion from color images , ranging from single view , multiple views and monocular video . We discuss them in the context of our work . Single-view Depth Estimation . While ill-posed , the emerging of deep learning technology enables the estimation of depth from a single color image . The early work directly formulates this into a per-pixel regression problem ( Eigen et al. , 2014 ) , and follow-up works improve the performance by introducing multi-scale network architectures ( Eigen et al. , 2014 ; Eigen & Fergus , 2015 ) , skipconnections ( Wang et al. , 2015 ; Liu et al. , 2016 ) , powerful decoder and post process ( Garg et al. , 2016 ; Laina et al. , 2016 ; Kuznietsov et al. , 2017 ; Wang et al. , 2015 ; Liu et al. , 2016 ) , and new loss functions ( Fu et al. , 2018 ) . Even though single view based methods generate plausible results , the models usually resort heavily to the prior in the training data and suffer from generalization capability . Nevertheless , these methods still act as an important component in some multi-view systems ( Tang & Tan , 2018 ) Traditional Structure-from-Motion Simultaneously estimating 3d structure and camera motion is a well studied problem which has a traditional tool-chain of techniques ( Furukawa et al. , 2010 ; Newcombe et al. , 2011 ; Wu et al. , 2011b ) . Structure from Motion ( SfM ) has made great progress in many aspects . Lowe ( 2004 ) ; Han et al . ( 2015 ) aim at improving features and Snavely ( 2011 ) introduce new optimization techniques . More robust structures and data representations are introduced by Gherardi et al . ( 2010 ) ; Schonberger & Frahm ( 2016 ) . Simultaneous Localization and Sapping ( SLAM ) systems track the motion of the camera and build 3D structure from video sequence ( Newcombe et al. , 2011 ; Engel et al. , 2014 ; Mur-Artal et al. , 2015 ; Mur-Artal & Tardós , 2017 ) . Engel et al . ( 2014 ) propose the photometric bundle adjustment algorithm to directly minimize the photometric error of aligned pixels . However , traditional SfM and SLAM methods are sensitive to low texture region , occlusions , moving objects and lighting changes , which limit the performance and stability . Deep Learning for Structure-from-Motion Deep neural networks have shown great success in stereo matching and Structure-from-Motion problems . Ummenhofer et al . ( 2017 ) ; Wang et al . ( 2017 ) ; Vijayanarasimhan et al . ( 2017 ) ; Zhou et al . ( 2017 ) regress depth map and camera pose di- rectly in a supervised manner or by introducing photometric constraints between depth and motion as a self-supervision signal . Such methods solve the camera motion as a regression problem , and the relation between camera motion and depth prediction is neglected . Recently , some methods exploit multi-view photometric or feature-metric constraints to enforce the relationship between dense depth map and the camera pose in network . The SE3 transformer layer is introduced by Teed & Deng ( 2018 ) , which uses geometry to map flow and depth into a camera pose update . Wang et al . ( 2018 ) propose the differentiable camera motion estimator based on the Direct Visual Odometry ( Steinbrücker et al. , 2011 ) . Clark et al . ( 2018 ) using a LSTM-RNN ( Hochreiter et al. , 2001 ) as the optimizer to solve nonlinear least squares in two-view SfM . Tang & Tan ( 2018 ) train a network to generate a set of basis depth maps and optimize depth and camera poses in a BA-layer by minimizing a feature-metric error . 3 ARCHITECTURE . Our framework receives frames of a scene from different viewpoints , and produces photo-metrically and geometrically consistent depth maps across all frames and the corresponding camera poses . Similar to BA , we also assume initial structures ( i.e depth maps ) and motions ( i.e . camera poses ) are given . Note that the initialization is not necessary to be super accurate for the good performance using our framework and thus can be easily obtained from some direct regression methods ( Ummenhofer et al. , 2017 ) . Now we introduce the detail of our model – DeepSFM . Without loss of generality , we describe our model taking two images as input , namely the reference image and the source image , as an example , and all the technical components can be extended for multiple images straightforward . As shown in Figure 1 , we first extract feature maps from input images through a shared encoder . We then sample the solution space for depth uniformly in the inverse-depth space between a predefined minimum and maximum range and camera pose around the initialization respectively . After that , we build cost volumes accordingly to reason the confidence of each hypothesis . This is achieved by validating the consistency between the feature of the reference view and the ones warped from the source image . Besides photo-metric consistency that measures the color image similarity , we also take into account the geometric consistency across warped depth maps . Note that depth and pose require different designs of cost volume to efficiently sample the hypothesis space . Gradients can back-propagate through cost volumes , and cost-volume construction does not affect any trainable parameters . The cost volumes are then fed into 3D CNN to regress new depth and pose . These updated value can be used to create new cost volumes , and the model improves the prediction iteratively . For notations , we denote { Ii } ni=1 as the image sequences in one scene , { Di } ni=1 as the corresponding ground truth depth maps , { Ki } ni=1 as the camera intrinsics , { Ri , ti } ni=1 as the ground truth rotations and translations of camera , { D∗i } ni=1 and { R ∗ i , t∗i } ni=1 as initial depth maps and camera pose parameters for constructing cost volumes , where n is the number of image samples . 3.1 2D FEATURE EXTRACTION . Given the input sequences { Ii } ni=1 , we extract the 2D CNN feature { Fi } ni=1 for each frame . Firstly , a 7 layers ’ CNN with kernel size 3×3 is applied to extract low contextual information . Then we adopt a spatial pyramid pooling ( SPP ) ( Kaiming et al. , 2014 ) module , which can extract hierarchical multiscale features through 4 average pooling blocks with different pooling kernel size ( 4×4 , 8×8 , 16× 16 , 32 × 32 ) . Finally , we pass the concatenated features through 2D CNNs to get the 32-channel image features after upsampling these multi-scale features into the same resolution . These image sequence features are used by the building of both our depth based and pose based cost volumes .

Summary: The authors propose a SfM model which integrates geometric consistency with a learned pose and depth network. An initial estimate of depth and pose are used to construct pose and depth cost volumes, which are then fed into a pose regression and depth refinement network, to produce a new set of cost volumes, and so on. In this manner, the pose and depth estimation are improved iteratively.