DPSNet: End-to-end Deep Plane Sweep Stereo

1 INTRODUCTION . Various image understanding tasks , such as semantic segmentation Couprie et al . ( 2013 ) and human pose/action recognition Shotton et al . ( 2011 ) ; Wang et al . ( 2016 ) , have been shown to benefit from 3D scene information . A common approach to reconstructing 3D geometry is by multiview stereo , which infers depth based on point correspondences among a set of unstructured images Hartley & Zisserman ( 2003 ) ; Schönberger et al . ( 2016 ) . To solve for these correspondences , conventional techniques employ photometric consistency constraints on local image patches . Such photo-consistency constraints , though effective in many instances , can be unreliable in scenes containing textureless and reflective regions . Recently , convolutional neural networks ( CNNs ) have demonstrated some capacity to address this issue by leveraging semantic information inferred from the scene . The most promising of these methods employ a traditional stereo matching pipeline , which involves computation of matching cost volumes , cost aggregation , and disparity estimation Flynn et al . ( 2016 ) ; Kendall et al . ( 2017 ) ; Huang et al . ( 2018 ) ; Chang & Chen ( 2018 ) . Some are designed for binocular stereo Ummenhofer et al . ( 2017 ) ; Kendall et al . ( 2017 ) ; Chang & Chen ( 2018 ) and can not readily be extended to multiple views . The CNN-based techniques for multiview processing Flynn et al . ( 2016 ) ; Huang et al . ( 2018 ) both follow the plane-sweep approach , but require plane-sweep volumes as input to their networks . As a result , they are not end-to-end systems that can be trained from input images to disparity maps . In this paper , we present Deep Plane Sweep Network ( DPSNet ) , an end-to-end CNN framework for robust multiview stereo . In contrast to previous methods that employ the plane-sweep approach Huang et al . ( 2018 ) ; Flynn et al . ( 2016 ) , DPSNet fully models the plane-sweep process , including construction of plane-sweep cost volumes , within the network . This is made possible through the use of a differentiable warping module inspired by spatial transformer networks Jaderberg et al . ( 2015 ) to build the cost volumes . With the proposed network , plane-sweep stereo can be learned in an end-to-end fashion . Additionally , we introduce a cost aggregation module based on ∗part of the work was done during an internship at Microsoft Research Asia local cost-volume filtering Rhemann et al . ( 2011 ) for context-aware refinement of each cost slice . Through this cost-volume regularization , the effects of unreliable matches scattered within the cost volume are reduced considerably . With this end-to-end network for plane-sweep stereo and the proposed cost aggregation , we obtain state-of-the-art results over several standard datasets . Ablation studies indicate that each of these technical contributions leads to appreciable improvements in reconstruction accuracy . 2 RELATED WORK . CNN-based depth estimation has been studied for stereo matching , depth from single images , and multiview stereo . Recent work in these areas are briefly reviewed in the following . Stereo matching Methods for stereo matching address the particular case of depth estimation where the input is a pair of rectified images captured by a stereo rig . Various network structures have been introduced for this problem . Zbontar & LeCun ( 2016 ) present a Siamese network structure to compute matching costs based on the similarity of two image patches . The estimated initial depth is then refined by traditional cost aggregation and refinement as post-processing . Mayer et al . ( 2016 ) directly stack several convolution and deconvolution layers upon the matching costs and train the network to minimize the distance between the estimates and ground truth . Liang et al . ( 2018 ) propose a CNN that estimates initial disparity and then refines it using both prior and posterior feature consistency in an end-to-end manner . Kendall et al . ( 2017 ) leverage geometric knowledge in building a cost volume from deep feature representations . It also enables learning of contextual information in a 3D volume and regresses disparity in an end-to-end manner . Chang & Chen ( 2018 ) introduce a pyramid pooling module for incorporating global contextual information into image features and a stacked hourglass 3D CNN to extend the regional support of contextual information . Depth from single images Similar to these stereo matching approaches , single-image methods extract CNN features to infer scene depths and perform refinements to increase depth accuracy . The first of these methods was introduced by Eigen et al . ( 2014 ) , which demonstrated that CNN features could be utilized for depth inference . Later , Liu et al . ( 2016 ) combined a superpixel-based conditional random field ( CRF ) to a CNN to improve the quality of depth estimates from single images . To facilitate training , recent studies Zhou et al . ( 2017 ) ; Wang et al . ( 2018 ) ; Mahjourian et al . ( 2018 ) ; Yin & Shi ( 2018 ) present an end-to-end learning pipeline that utilizes the task of view synthesis as supervision for single-view depth and camera pose estimation . These systems consist of a depth network and a pose estimation network which simultaneously train on sequential images with a loss computed from images warped to nearby views using the estimated depth . View synthesis has similarly been used as supervision by warping between stereo image pairs Garg et al . ( 2016 ) ; Godard et al . ( 2017 ) . In contrast to these single-image works which employ warping as a component of view synthesis for self-supervised learning , our network computes warps with respect to multiple depth planes to produce plane-sweep cost volumes both for training and at test time . The cost volumes undergo further processing in the form of cost aggregation and regularization to improve the robustness of depth estimates . Multi-view stereo In multi-view stereo , depth is inferred from multiple input images acquired from arbitrary viewpoints . To solve this problem , some methods recover camera motion between the unstructured images but are designed to handle only two views Ummenhofer et al . ( 2017 ) ; Li et al . ( 2018 ) . The DeMoN system Ummenhofer et al . ( 2017 ) consists of encoder-decoder networks for optical flow , depth/motion estimation , and depth refinement . By alternating between estimating optical flow and depth/motion , the network is forced to use both images in estimating depth , rather than resorting to single-image inference . Li et al . ( 2018 ) perform monocular visual odometry in an unsupervised manner . In the training step , the use of stereo images with extrinsic parameters allows 3D depth estimation to be estimated with metric scale . Among networks that can handle an arbitrary number of views , camera parameters are assumed to be known or estimated by conventional geometric methods . Ji et al . ( 2017 ) introduce an endto-end learning framework based on a viewpoint-dependent voxel representation which implicitly encodes images and camera parameters . The voxel representation restricts the scene resolution that can be processed in practice due to limitations in GPU memory . Im et al . ( 2018b ) formulate a geometric relationship between optical flow and depth to refine the estimated scene geometry , but is designed for image sequences with a very small baseline , i.e. , an image burst from a handheld camera . Huang et al . ( 2018 ) compute a set of plane-sweep volumes using calibrated pose data as input for the network , which then predicts an initial depth feature using an encoder-decoder network . In the depth prediction step , they concatenate a reference image feature to the decoder input as an intra-feature aggregation , and cost volumes from each of the input images are aggregated by max-pooling to gather information for the multiview matching . Its estimated depth map is refined using a conventional CRF . By contrast , our proposed DPSNet is developed to be trained end-to-end from input images to the depth map . Moreover , it leverages conventional multiview stereo concepts by incorporating context-aware cost aggregation . Finally , we would like to refer the reader to the concurrent work by Yao et al . ( 2018 ) that also adopts differential warping to construct a multi-scale cost volume , then refined an initial depth map guided by a reference image feature . Our work is independent of this concurrent effort . Moreover , we make distinct contributions : ( 1 ) We focus on dense depth estimation for a reference image in an end-to-end learning manner , different from Yao et al . ( 2018 ) which reconstructs the full 3D of objects . ( 2 ) Our cost volume is constructed by concatenating input feature maps , which enables inference of accurate depth maps even with only two-view matching . ( 3 ) Our work refines every cost slice by applying context features of a reference image , which is beneficial for alleviating coarsely scattered unreliable matches such as for large textureless regions . 3 APPROACH . Our Deep Plane Sweep Network ( DPSNet ) is inspired by traditional multiview stereo practices for dense depth estimation and consists of four parts : feature extraction , cost volume generation , cost aggregation and depth map regression . The overall framework is shown in Figure 2 . 3.1 MULTI-SCALE FEATURE EXTRACTION . We first pass a reference image and target images through seven convolutional layers ( 3 × 3 filters except for the first layer , which has a 7×7 filter ) to encode them , and extract hierarchical contextual information from these images using a spatial pyramid pooling ( SPP ) module He et al . ( 2014 ) with four fixed-size average pooling blocks ( 16 × 16 , 8 × 8 , 4 × 4 , 2 × 2 ) . The multi-scale features extracted by SPP have been shown to be effective in many visual perception tasks such as visual recognition He et al . ( 2014 ) , scene parsing Zhao et al . ( 2017 ) and stereo matching Huang et al . ( 2018 ) . After upsampling the hierarchical contextual information to the same size as the original feature map , we concatenate all the feature maps and pass them through 2D convolutional layers . This process yields 32-channel feature representations for all the input images , which are next used in building cost volumes . 3.2 COST VOLUME GENERATION USING UNSTRUCTURED TWO-VIEW IMAGES . We propose to generate cost volumes for the multiview images by adopting traditional plane sweep stereo Collins ( 1996 ) ; Yang & Pollefeys ( 2003 ) ; Ha et al . ( 2016 ) ; Im et al . ( 2018a ) , originally devised for dense depth estimation . The basic idea of plane sweep stereo is to back-project the image set onto successive virtual planes in the 3D space and measure photo-consistency among the warped images for each pixel . In a similar manner to traditional plane sweep stereo , we construct a cost volume from an input image pair . To reduce the effects of image noise , multiple images can be utilized by averaging cost volumes for other pairs . For this cost volume generation network , we first set the number of virtual planes perpendicular to the z-axis of the reference viewpoint [ 0 , 0 , 1 ] ᵀ and uniformly sample them in the inverse-depth space as follows : dl = ( L× dmin ) l , ( l = 1 , .. , L ) , ( 1 ) where L is the total number of depth labels and dmin is the minimum scene depth as specified by the user . Then , we warp all the paired features Fi , ( i = 1 , .. , N ) , where i is an index of viewpoints and N is the total number of input views , into the coordinates of the reference feature ( of size Width × Height × CHannel ) using pre-computed intrinsics K and extrinsic parameters consisting of a rotation matrix Ri and a translation matrix ti of the ith camera : F̃il ( u ) = Fi ( ũl ) , ũl ∼ K [ Ri|ti ] [ ( K−1u ) dl 1 ] , ( 2 ) where u , ũl are the homogeneous coordinates of a pixel in the reference view and the projected coordinates onto the paired view , respectively . F̃il ( u ) denotes the warped features of the paired image through the lth virtual plane . Unlike the traditional plane sweeping method which utilizes a distance metric , we use a concatenation of features in learning a representation and carry this through to the cost volume as proposed in Kendall et al . ( 2017 ) . We obtain a 4D volume ( W×H×2CH×L ) by concatenating the reference image features and the warped image features for all of the depth labels . In Eq . ( 2 ) , we assume that all images are captured by the same camera , but it can be directly extended to images with different intrinsics . For the warping process , we use a spatial transformer network Jaderberg et al . ( 2015 ) for all hypothesis planes , which does not require any learnable parameters . In Table 3 , we find that concatenating features improves performance over the absolute difference of the features . Given the 4D volume1 , our DPSNet learns a cost volume generation of size W ×H × L by using a series of 3D convolutions on the concatenated features . All of the convolutional layers consist of 3 × 3 × 3 filters and residual blocks . In the training step , we only use one paired image ( while the other is the reference image ) to obtain the cost volume . In the testing step , we can use any number of paired images ( N ≥ 1 ) by averaging all of the cost volumes .

Summary: The paper describes a method for learning a deep neural network for multi-view stereo. The overall network includes feature-extraction layers applied to all images, followed by a spatial-transformer network (which is differentiable, but with no learnable parameters) that is applied to warp these features from every matching image to the reference image's co-ordinate frame for a series of candidate depth planes, followed by concatenation of the reference and match image features and 3D convolution layers to form a cost volume. The cost volumes of different pairs are averaged, and additional layers are used to refine this cost volume while relying on the reference image's RGB features, followed by soft-max and an expectation over depth values to output the final depth at each pixel. The entire network is trained end-to-end and experiments show that it outperforms state-of-the-art methods for MVS by a significant margin on a number of datasets.