CURVATURE-GUIDED DYNAMIC SCALE NETWORKS FOR MULTI-VIEW  STEREO

1 INTRODUCTION . A challenging problem in multi-view stereo ( MVS ) is accurately estimating dense correspondences across a collection of high-resolution images . Many MVS studies have tried to solve ill-posed MVS problems such as matching ambiguity and high computational complexity . In such problems , learning-based MVS methods ( Chen et al. , 2019 ; Luo et al. , 2019 ; Chen et al. , 2020a ; Xu & Tao , 2020a ; Xue et al. , 2019 ) usually outperform traditional methods ( Galliani et al. , 2015 ; Schonberger & Frahm , 2016 ) . The learning-based methods are generally composed of three steps , including feature extraction , cost volume formulation , and cost volume regularization . Most studies made an effort to improve the performance of cost formulation ( Luo et al. , 2019 ; Zhang et al. , 2020 ; Xu & Tao , 2020c ; Yi et al. , 2020 ) or cost regularization ( Luo et al. , 2019 ; 2020 ; Wang et al. , 2021 ) . The visibility information ( Zhang et al. , 2020 ; Xu & Tao , 2020c ) or attention techniques ( Yi et al. , 2020 ; Luo et al. , 2020 ) are considered for cost aggregation and feature matching , respectively . Furthermore , some studies employed a hybrid 3D UNet structure ( Luo et al. , 2019 ; Sormann et al. , 2020 ) for cost volume regularization . While these approaches significantly improve MVS performances , their feature extraction networks have some drawbacks that degrade the quality of 3D reconstruction . First , they observed a restricted size of receptive fields from the fixed size of convolutional kernels . This leads to difficulty in learning a robust pixel-level representation when an object ’ s scale varies extremely in images . As a result , the extracted features cause the low quality of matching costs , especially when the difference of camera poses between reference and source images is large . Second , MVS networks are often trained on low-resolution images because of limited memory and restricted computation time . Therefore , the existing feature extraction methods utilizing fixed-scale feature representation on deep networks could not generalize to high-resolution images in prediction ; the MVS performance is downgraded on a high-resolution image set . To address these issues , we propose a new feature extraction network that can be adapted to various object scales and image resolutions . We refer the proposed network as curvature-guided dynamic scale feature network ( CDSFNet ) . The proposed network is composed of novel convolution layers , curvature-guided dynamic scale convolution ( CDSConv ) , which select a suitable patch scale for each pixel to learn robust representation . This selection is achieved by computing the normal curvature of the image surface at several candidate scales and then analyzing the computed outputs via a classification network . The pixel-level patch scales estimated from the proposed network are dynamic with respect to textures , scale of objects , and epipolar geometry . Therefore , it trains more discriminative features than existing networks ( Gu et al. , 2020 ; Cheng et al. , 2020 ; Yang et al. , 2020 ; Zhang et al. , 2020 ; Luo et al. , 2020 ) for accurate matching cost computation . Fig . 1a illustrates the dynamic pixel-level patch scales estimated from CDSConv where the window size is ranged from 7×7 to 17×17 . As shown in the figure , CDSConv predicts the adaptive patch scales where the scale is small in thin structure or texture-rich regions and large in the texture-less regions . Moreover , the scale is dynamic according to image resolution . CDSConv generally produces small scale values in a low-resolution image and large scale values in a high-resolution image . We also formulate a new MVS framework , CDS-MVSNet , which aims at improving the quality of MVS reconstruction while decreasing computation time and memory consumption . Similar to previous works ( Gu et al. , 2020 ; Cheng et al. , 2020 ; Yang et al. , 2020 ; Wang et al. , 2021 ) , CDS-MVSNet is composed of a cascade network structure to estimate high-resolution depth maps in a coarse-tofine manner . For each cascade stage , it formulates a 3D cost volume based on the output features of CDSFNet , which reduces the matching ambiguity by using proper pixel ’ s scale . For instance , consider a 3D point on the scene that needs to be reconstructed , CDSFNet can capture the same context information in both reference and source views for this point to extract the matchable features . Furthermore , CDS-MVSNet applies visibility-based cost aggregation to improve the performance of stereo matching . The pixel-wise visibility is estimated from the curvature information that encodes the matching ability of extracted features implicitly . Therefore , CDS-MVSNet performs accurate stereo matching and cost aggregation even on low-resolution images . Finally , the proposed network can generate high quality 3D models only by processing half-resolution images . This approach can significantly reduce the computation time and memory consumption of MVS reconstruction . The contribution of this paper is summarized as follows : • We present the CDSConv that learns dynamic scale features guided by the normal curvature of image surface . This operation is implemented by approximating surface normal curvatures in several candidate scales and choosing a proper scale via a classification network . • We propose a new feature extraction network , CDSFNet , which is composed of multiple CDSConv layers for learning robust representation at pixel level . CDFSNet estimates the optimal pixel ’ s scale to learn features . The scale is selected adaptively with respect to structures , textures , and epipolar constraints . • We present CDS-MVSNet for MVS depth estimation . CDS-MVSNet performs accurate stereo matching and cost aggregation by handling the ambiguity and visibility in the image matching process . It also significantly reduces the run-time and memory consumption by processing the half-resolution images while maintaining the reconstruction quality . • We verify the effectiveness of our method on two benchmark datasets ; DTU ( Aanæs et al. , 2016 ) and Tanks & Temples ( Knapitsch et al. , 2017 ) . The results demonstrate that the proposed feature learning method boosts the performance of MVS reconstruction . 2 RELATED WORK . Multi-view Stereo . Traditional MVS studies can be divided into three categories : volumetric , point cloud-based , and depth map-based methods ( Furukawa & Hernández , 2015 ) . Comparatively , depth map-based methods are more concise and flexible , they estimate depth maps for all reference images and then perform depth fusion to achieving a 3D model ( Galliani et al. , 2015 ; Schönberger et al. , 2016 ; Xu & Tao , 2019 ) . The popular MVS methods in this category are usually based on PatchMatch Stereo algorithm ( Barnes et al. , 2009 ) . Recently , many studies have applied a learning-based approach , which has achieved significant performance improvements over traditional methods . Several volumetric methods ( Kar et al. , 2017 ; Ji et al. , 2017 ) employed 3D CNNs to explicitly predict global 3D surfaces . Yao et al . ( 2018 ) proposed a depth-map-based method , MVSNet , which is an end-to-end trainable architecture with three steps . However , these methods still confronted the matching ambiguity problem and could not process high-resolution images due to limited memory . Many methods based on the three-step architecture of MVSNet are proposed to address these issues . Several studies ( Gu et al. , 2020 ; Cheng et al. , 2020 ; Yang et al. , 2020 ; Wang et al. , 2021 ; Zhang et al. , 2020 ) applied a coarse-to-fine framework which estimates the depths through multiple stages to reduce the computational complexity . They used the estimated depth of each stage to adjust the depth hypothesis range for cost formulation in the next stage . To enhance the matching cost volume , other methods introduced a novel strategy for cost volume formulation . Luo et al . ( 2019 ) proposed a patch-wise feature matching instead of pixel-wise matching to compute the cost . Yi et al . ( 2020 ) and Luo et al . ( 2020 ) applied an attention technique to improve feature matching . Zhang et al . ( 2020 ) ; Xu & Tao ( 2020c ) estimated visibility information to score the view weights for cost aggregation . All these methods ( Gu et al. , 2020 ; Cheng et al. , 2020 ; Yang et al. , 2020 ; Wang et al. , 2021 ; Zhang et al. , 2020 ; Luo et al. , 2020 ; Yu & Gao , 2020 ) had a trade-off between computation and performance ; the methods with high performance usually require expensive computations . Also , they only applied standard CNNs with static scale representation for feature extraction . In this work , we aim to learn dynamic scale representation for more accurate feature matching . We also estimate pixel-wise visibility information to remove noises and wrong matching pixels in cost aggregation . These goals are achieved by analyzing the normal curvature of the image surface . Recently , Xu et al . ( 2020 ) proposed an MVS method that considers the normal curvature information for feature extraction . This method handles ambiguity in the patch-match stereo method ( Barnes et al. , 2009 ) . It selects a patch scale for each pixel by thresholding the normal curvatures computed in several candidate scales . Although the method effectively improves the quality of patch matching , the use of its handcrafted features limits the overall MVS performance . In contrast , our method does not require extracting handcrafted features and determining the threshold for scale selection . Furthermore , CDSConv operates not only on the color information of images but also on the high dimensional features . Multiple CDSConv operations can be stacked to form a very deep architecture for robust feature extraction in MVS . Multi-scale features . Due to large-scale variations of objects and textures in complex scenes , multiscale representations are adapted for accurate and effective dense pixel-level prediction in many computer vision tasks . For the MVS task , Feature Pyramid Network ( FPN ) ( Lin et al. , 2017 ; Gu et al. , 2020 ; Cheng et al. , 2020 ; Zhang et al. , 2020 ; Luo et al. , 2020 ) is a common approach for multi-scale feature extraction . It is designed as the UNet-liked architecture ( Ronneberger et al. , 2015 ) to fuse features from different scales into a single one . However , the multi-scale of FPN is achieved at an image level through down-upsampling operations . It can not capture the large-scale variation of objects at a pixel level . Several studies in image segmentation can learn pixel-level multi-scale features by using multiple kernels with different sizes ( He et al. , 2019 ; Liu et al. , 2016 ; Chen et al. , 2017 ; Zhao et al. , 2017 ) . However , these methods prefer a large receptive field for segmentation tasks and usually ignore the details of objects essential for MVS tasks . In contrast to the aforementioned methods ( Gu et al. , 2020 ; Cheng et al. , 2020 ; Zhang et al. , 2020 ; Luo et al. , 2020 ; He et al. , 2019 ) , our proposed CDSConv can select an optimal scale for each pixel to produce a robust representation and enhance the matching uncertainty . Moreover , our feature extraction CDSFNet can cover denser scales by utilizing the power of a deep CNN . Dynamic filtering . Instead of observing static receptive fields , several studies modified standard convolution to choose the adaptive receptive field for each pixel ( Wu et al. , 2018 ; Jia et al. , 2016 ; Han et al. , 2018 ) . SAC ( Zhang et al. , 2017 ) was proposed to modify the fixed-size receptive field by learning position-adaptive scale coefficients . Deformable ConvNets ( Dai et al. , 2017 ; Zhu et al. , 2019 ) learned offsets for each pixel in a regular sampling grid of standard convolution to enlarge the sampling field with an arbitrary form that can discover geometric-invariant features . Recently , Chen et al . ( 2020b ) proposed a dynamic convolution composed of multiple kernels with the same size to increase model complexity without increasing the network depth or width . All these existing works learn dynamic filters directly from the input images . They are suitable for segmentation or recognition tasks . For MVS task , they ignore the epipolar constraint between reference and source images . In contrast , our dynamic filtering is guided by normal curvature computed in the direction of epipolar line , which can measure the matching capacity between reference and source image patches . To the best of our knowledge , this is the first work that introduces dynamic filters to MVS .

Summary: This paper proposes a method named CDSFNet for the multi-view stereo problem. The network is composed of curvature-guided dynamic scale convolution (CDSConv) layers to first estimate approximate normal curvature at different candidate scales, and then do a classification to find the optimal scale. The module is then embedded with cascade structure of CasMVSNet, to reduce the matching ambiguity by considering the proper pixel scales. The method is evaluated on DTU, BlendedMVS and Tanks & Temples datasets, which archives state-of-the-art performance compared with other related methods.