DeepV2D: Video to Depth with Differentiable Structure from Motion

1 INTRODUCTION . In video to depth , the task is to estimate depth from a video sequence . The problem has traditionally been approached using Structure from Motion ( SfM ) , which takes a collection of images as input , and jointly optimizes over 3D structure and camera motion ( Schonberger & Frahm , 2016b ) . The resulting camera parameter estimates can be used as input to Multi-View Stereo in order to build a more complete 3D representation such as surface meshes and depth maps ( Furukawa et al. , 2015 ; Furukawa & Ponce , 2010 ) . In parallel , deep learning has been highly successful in a number of 3D reconstruction tasks . In particular , given ground truth depth , a network can learn to predict depth from a single image ( Eigen et al. , 2014 ; Eigen & Fergus , 2015 ; Laina et al. , 2016 ) , stereo images ( Kendall et al. , 2017 ; Mayer et al. , 2016a ) , or collections of frames ( Zhou et al. , 2018 ; Kar et al. , 2017 ; Tang & Tan , 2018 ; Yao et al. , 2018 ) . One advantage of deep networks is that they can use single-image cues such as texture gradients and shading as shown by their strong performance on depth estimation from a single image ( Eigen et al. , 2014 ; Eigen & Fergus , 2015 ; Laina et al. , 2016 ) . Furthermore , differentiable network modules can be composed so that entire pipelines ( i.e . feature extraction , feature matching , regularization ) can be learned directly from training data . On the other hand , as recent work has shown , it is often hard to train generic network layers to directly utilize multiview geometry ( e.g . using interframe correspondence to recover depth ) , and it is often advantageous to embed knowledge of multiview geometry through specially designed layers or losses ( Ummenhofer et al. , 2017 ; Kendall & Cipolla , 2017 ; Zhou et al. , 2017 ; Vijayanarasimhan et al. , 2017 ; Zhou et al. , 2018 ) . In this work , we continue the direction set forth by recent works ( Ummenhofer et al. , 2017 ; Kendall et al. , 2017 ; Tang & Tan , 2018 ; Zhou et al. , 2018 ; Kar et al. , 2017 ; Wang et al. , 2018 ) that combine the representation ability of neural networks with the geometric principles underlying image forma- tion . We propose DeepV2D , a composition of classical geometrical algorithms which we turn into differentiable network modules and combine into an end-to-end trainable architecture . DeepV2D interleaves two stages : camera motion estimation and depth estimation ( Figure 1 ) . The motion module takes depth as input , and outputs an incremental update to camera motion . The depth module takes camera motion as input , and performs stereo reconstruction to predict depth . At test time , DeepV2D acts as block coordinate descent , alternating between updating depth and camera motion . To estimate camera motion we introduce Flow-SE3 , a new motion estimation architecture , which outputs an incremental update to camera motion . Flow-SE3 takes depth as input , and estimates dense 2D correspondence between pairs of frames . We unroll a single iteration of Perspectiven-Point ( PnP ) ( Lepetit et al. , 2009 ; Li et al. , 2012 ) performing Gauss-Newton updates over SE3 perturbations to minimize geometric reprojection error . The new estimate of camera motion can then be fed back into Flow-SE3 , which re-estimates correspondence for a finer grain pose update . Our Depth Module builds upon prior work ( Kendall et al. , 2017 ; Yao et al. , 2018 ) and formulates multiview-stereo ( MVS ) reconstruction as a single feed-forward network . Like classical MVS , we leverage geometry to build a cost volume over video frames , but use trainable network for both feature extraction and matching . Our work shares similarities with prior works ( Ummenhofer et al. , 2017 ; Kendall et al. , 2017 ; Tang & Tan , 2018 ; Zhou et al. , 2018 ; Kar et al. , 2017 ; Wang et al. , 2018 ) that also combine deep learning and multiview geometry , but is novel and unique in that it essentially “ differentializes ” a classical SfM pipeline that alternates between stereopsis , dense 2D feature matching , and PnP . As a comparison , DeMon ( Ummenhofer et al. , 2017 ) and DeepTAM ( Zhou et al. , 2018 ) differentialize stereopsis and feature matching , but not PnP because they use a generic network to predict camera motion . Another comparison is with BA-Net ( Tang & Tan , 2018 ) , whose classical analogue is performing bundle adjustment from scratch to optimize feature alignment over camera motion and the coefficients of a limited set of depth maps ( depth basis ) . In other words , BA-Net performs one joint nonlinear optimization over all variables , whereas we decompose the joint optimization into more tractable subproblems and do block coordinate descent . Our decomposition is more expressive in terms of reconstruction since we can optimize directly over per-pixel depth and are not constrained by a depth basis , which can potentially limit the accuracy of the final depth . In our experiments , we demonstrate the effectiveness of DeepV2D across a variety of datasets and tasks , and outperform strong methods such as DeepTAM ( Zhou et al. , 2018 ) , DeMoN ( Ummenhofer et al. , 2017 ) , BANet ( Tang & Tan , 2018 ) , and MVSNet ( Yao et al. , 2018 ) . As we show , alternating depth and motion estimation quickly converges to good solutions . On all datasets we outperform all existing single-view and multi-view approaches . We also show superior cross-dataset generalizability , and can outperform existing methods even when training on entirely different datasets . 2 RELATED WORK . Structure from Motion : Beginning with early systems designed for small image collections ( Longuet-Higgins , 1981 ; Mohr et al. , 1995 ) , Structure from Motion ( SfM ) has improved dramatically in regards to robustness , accuracy , and scalability . Advances have come from improved features ( Lowe , 2004 ; Han et al. , 2015 ) , optimization techniques ( Snavely , 2009 ) , and more scalable data structures and representations ( Schonberger & Frahm , 2016a ; Gherardi et al. , 2010 ) , culminating in a number of robust systems capable of large-scale reconstruction task ( Schonberger & Frahm , 2016a ; Snavely , 2011 ; Wu et al. , 2011 ) . Ranftl et al . ( 2016 ) showed that SfM could be extended to reconstruct scenes containing many dynamically moving objects . However , SfM is limited by the accuracy and availability of correspondence . In low texture regions , occlusions , or lighting changes SfM can produce noisy or missing reconstructions . Simultaneous Localization and Mapping ( SLAM ) jointly estimates camera motion and 3D structure from a video sequence ( Engel et al. , 2014 ; Mur-Artal et al. , 2015 ; Mur-Artal & Tardós , 2017 ; Newcombe et al. , 2011 ; Engel et al. , 2018 ) . LSD-SLAM ( Engel et al. , 2014 ) is unique in that it relies on a featureless approach to 3D reconstruction , directly estimating depth maps and camera pose by minimizing photometric error . Our Motion Network behaves similarly to the tracking component in LSD-SLAM , but we use a network which predicts misalignment directly instead of using intensity gradients . We end up with an easier optimization problem characteristic of indirect methods ( Mur- Artal et al. , 2015 ) , while retaining the flexibility of direct methods in modeling edges and smooth intensity changes ( Engel et al. , 2018 ) . Geometry and Deep Learning : Geometric principles has motivated the design of many deep learning architectures . In video to depth , we need to solve two subproblems : depth estimation and motion estimation . Depth : End-to-end networks can be trained to predict accurate depth from a rectified pair of stereo images ( Han et al. , 2015 ; Mayer et al. , 2016a ; Kendall et al. , 2017 ; Chang & Chen , 2018 ) . Kendall et al . ( 2017 ) and Chang & Chen ( 2018 ) design network architectures specifically for stereo matching . First , they apply a 2D convolutional network to extract learned features , then build a cost volume over the learned features . They then apply 3-D convolutions to the cost volume to perform feature matching and regularization . A similar idea has been extended to estimate 3D structure from multiple views ( Kar et al. , 2017 ; Yao et al. , 2018 ) . In particular , MVSNet ( Yao et al. , 2018 ) estimates depth from multiple images . However , these works require known camera poses as input , while our method estimates depth from a video where the motion of the camera is unknown and estimated during inference . Motion : Several works have used deep networks to predict camera pose . Kendall et al . ( 2015 ) focus on the problem of camera localization , while other work ( Zhou et al. , 2017 ; Vijayanarasimhan et al. , 2017 ; Wang et al. , 2017 ) propose methods which estimate camera motion between a pairs of frames in a video . Networks for motion estimation have typically relied on generic network components whereas we formulate motion estimation as a least-squares optimization problem . Whereas prior work has focused on estimating relative motion between pairs of frames , we can jointly update the pose of a variable number of frames . Depth and Motion : Geometric information has served as a self-supervisory signal for many recent works ( Vijayanarasimhan et al. , 2017 ; Zhou et al. , 2017 ; Wang et al. , 2018 ; Yin & Shi , 2018 ; Yang et al. , 2018 ; Godard et al. , 2017 ; Mahjourian et al. , 2018 ) . In particular , Zhou et al . ( 2017 ) and Vijayanarasimhan et al . ( 2017 ) trained a single-image depth network and a pose network while supervising on photometric consistency . However , while these works use geometric principles for training , they do not use multiple frames to predict depth at inference . DeMoN ( Ummenhofer et al. , 2017 ) and DeepTAM ( Zhou et al. , 2018 ) where among the first works to combine motion estimation and multi-view reconstruction into a trainable pipeline . DeMoN ( Ummenhofer et al. , 2017 ) operates on two frames and estimates depth and motion in separate network branches , while DeepTAM ( Zhou et al. , 2018 ) can be used on variable number of frames . Like our work and other classical SLAM framesworks ( Engel et al. , 2014 ; Newcombe et al. , 2011 ) , DeepTAM separates depth and motion estimation , however we maintain end-to-end differentiablity between our modules . A major innovation of DeepTAM was to formulate camera motion estimation in the form of incremental updates . In each iteration , DeepTAM renders the keyframe from a synthetic viewpoint , and predicts the residual motion from the rendered viewpoint and the target frame . Estimating depth and camera motion can be naturally modeled as a non-linear least squares problem , which has motivated several works to include an differentiable optimization layer within network architectures ( Tang & Tan , 2018 ; Wang et al. , 2018 ; Clark et al. , 2018 ; Bloesch et al. , 2018 ) . We follow this line of work , and propose the Flow-SE3 module which introduces a direct mapping from 2D correspondence to a 6-dof camera motion update . Our Flow-SE3 module is different from prior works such as DeMon ( Ummenhofer et al. , 2017 ) and DeepTAM ( Zhou et al. , 2018 ) which do not impose geometric constraints on camera motion and use generic layers . BA-Net ( Tang & Tan , 2018 ) and LS-Net ( Clark et al. , 2018 ) include optimization layers , but instead optimize over photometric error ( either pixel alignment ( Clark et al. , 2018 ) or feature alignment ( Tang & Tan , 2018 ) ) . Our Flow-SE3 module still imposes geometric constraints on camera motion like BA-Net ( Tang & Tan , 2018 ) , but we show that in minimizing geometric reprojection error ( difference of pixel locations ) , we end up with a well-behaved optimization problem , well-suited for end-to-end training . An important difference between our approach and BA-Net is that BA-Net performs one joint optimization problem by formulating Bundle-Adjustment as a differentiable network layer , whereas we separate motion and depth estimation . With this separation , we avoid the need for a depth basis . Our final reconstructed depth is the product of a cost volume , which can adapt the reconstruction as camera motion updates improve , while the output of BA-Net is restricted by the initial quality of the depth basis produced by a single-image network .

Summary: This work proposes a neural network architecture for joint depth and camera motion estimation on video sequences. The authors propose an architecture that incorporates classic principles from SfM, namely depth computation based on cost volumes and motion estimation based on the reprojection error of features. Extensive experiments on a variety of datasets are shown and support that this approach provides strong results.