A shallow feature extraction network with a large receptive field for stereo matching tasks

1 INTRODUCTION . Since the introduction of deep learning in the computer vision field , increasing the network depth ( that is , the number of layers in the network ) seems to be a necessary means to improve the feature extraction ability . Taking the object classification task as an example , as the network depth increases from the 8-layer network AlexNet ( Krizhevsky et al. , 2012 ) to the 16-layer network VGG ( Simonyan & Zisserman , 2014 ) and to the 101-layer network ResNet ( He et al. , 2015 ) , the classification accuracy constantly improves . There are two purposes of the deep network . First , the deep network can improve the ability to extract abstract features ( Zeiler & Fergus , 2013 ) , which are important for some vision tasks , such as object detection ( Girshick , 2015 ; Ren et al. , 2017 ) and classification . For example , for objects such as cups , their colors , shapes and sizes may be different , and they can not be accurately identified using only these primary feature information . Therefore , the feature extraction network must have the ability to extract more abstract semantic information . Second , the deep feature extraction network can obtain a larger receptive field to learn more context information ( Luo et al. , 2017 ; Liu et al. , 2018 ) . With the increase in the number of network layers , the size of the receptive field is also constantly increasing . In particular , after image sampling using a pooling operation , even the 3 * 3 convolution kernel has the ability to extract context information . Many studies ( Zeiler & Fergus , 2013 ; Yu & Koltun , 2016 ) have shown that the lower part of the convolution neural network mainly extracts primary features , such as the edges and corners , while the higher part can extract more abstract semantic information . However , many basic vision tasks rely more on basic feature information instead of the high-level abstract features . Stereo matching is one of the basic vision tasks . In the traditional stereo matching algorithm ( Scharstein & Szeliski , 2002 ) , the color similarity metrics of pixels are usually used to calculate the matching costs between the left and right images to find the matching points in the two images . After the introduction of deep learning , more robust feature information can be obtained through training and learning , which can effectively improve the performance of the stereo matching algorithm . At present , many excellent stereo matching algorithms based on deep learning , such as the GC-Net ( Kendall et al. , 2017 ) , PSMNet ( Chang & Chen , 2018 ) and GwcNet ( Guo et al. , 2019 ) , generally adopt similar processes , including feature extraction , matching cost volume construction , 3D convolution and disparity regression . This paper focuses on the feature extraction steps . The stereo matching task has two requirements for the feature extraction network . The first requirement is the enlargement of the receptive field as far as possible so that the network can obtain more context information , which is critical to solving the mismatching problems in the discontinuous disparity area . Because a larger receptive field can learn the relationships between different objects , even if there are problems , such as conclusion or inconsistent illumination , the network can use the context information to infer disparity and improve the stereo matching accuracy in the ill-posed regions . The second requirement is the maintenance of more details of the spatial structure , which can improve the matching accuracy of many small structures , such as railings , chains , traffic signs and so on . The existing feature extraction networks usually use a deep convolution neural network to obtain a larger receptive field and extract more abstract semantic information . In this process , with the increase of the network layers and the compression of the image size , substantial detailed information of the spatial structure is inevitably lost . We believe that compared with the abstract semantic information that is extracted by a deep network , the detailed information of the spatial structure is more important to improving the stereo matching accuracy . Based on this point of view , this paper proposes a novel structure of feature extraction network – a shallow feature extraction network . Unlike the common feature extraction network ( with ResNet-50 as the backbone ) , in this paper , the backbone of the feature extraction network only has 3 convolution layers , and the image is only downsampled once in the first convolution layer to compress the size of the image . This structure retains more details of the spatial structure and pays more attention to primary features such as the edges and corners of objects , while abandoning more abstract semantic information . To solve the problem that the size of the receptive field of the shallow structure is limited , this paper introduces the atrous spatial pyramid pooling ( ASPP ) module ( Chen et al. , 2018 ) . The ASPP module uses the dilated convolution to increase the receptive field size without increasing the number of parameters . In addition , the convolution layers with different dilation rate can obtain feature maps with multiscale receptive fields . The large receptive fields can be used to obtain context information and to solve the problem of mismatching in ill-posed regions , and the small receptive fields can be used to retain more detailed information of the spatial structure and to improve the stereo matching accuracy in local areas . To integrate feature maps with multiscale receptive fields , this paper designs the feature fusion module and introduces the channel attention mechanism ( Jie et al. , 2017 ) . We assign different weights to feature maps with different dilation rates in the channel dimensions . The weights are acquired through learning , and more weight and attention are given to the feature channels with greater roles . The advantages of a shallow feature extraction network with a large receptive field are twofold . One advantage is that the network can meet the two requirements of the stereo matching task for the feature extraction network . On the basis of ensuring the large receptive field , more details of the spatial structure are retained . The other advantage is that the network greatly reduces the number of parameters and the difficulties of network training and deployment . The feature extraction network that is designed in this paper is used to replace the feature extraction part of the existing stereo matching network , and state-of-the-art performance is achieved on the KITTI2015 dataset ( Geiger , 2012 ) . Compared with the reference network , the number of parameters is reduced by 42 % , and the matching accuracy is improved by 1.9 % . The main contributions of this paper are as follows . • A shallow feature extraction network is proposed to extract and retain more details of the spatial structure . This network can improve the stereo matching accuracy with fewer parameters . • The dilated convolution and ASPP module are introduced to enlarge the receptive field . We verify the effect of the dilated convolution on the receptive field using mathematics and experiments . • A feature fusion module , which integrates the feature maps with multiscale receptive fields , is designed and realizes the mutual complementary feature information of different scales . 2 RELATED WORK . In recent years , deep learning methods have gradually replaced traditional algorithms and become the mainstream stereo matching algorithm . The GC-Net ( Kendall et al. , 2017 ) designed a new stereo matching algorithm process based on deep learning , including feature extraction , matching cost volume construction , 3D convolution and disparity regression . First , in the feature extraction part , two deep convolution neural network with shared weights are used to extract the feature information from the left and right images . The matching cost volume is formed by cascading the left and right feature maps . Then , the 3D convolution is carried out on the matching cost volume , which can extract the feature representations from the three dimensions of height , width and disparity . Finally , the regression method is used to obtain the disparity map . Since the introduction of GC-Net , most stereo matching algorithms follow the stereo matching process of GC-Net . Focused on the feature extraction part , this section introduces many improved schemes for feature extraction networks using excellent algorithms in recent years . The PSMNet ( Chang & Chen , 2018 ) further deepened the feature extraction network structure , which took ResNet-50 as the backbone , and used the spatial pyramid pooling ( SPP ) module ( Kaiming et al. , 2014 ) to obtain the feature information at different scales . GwcNet ( Guo et al. , 2019 ) retained the backbone structure in the PSMNet , but it eliminated the SPP module , and proposed a new method to form the matching cost volume using the group-wise correlation . Based on the PSMNet , MCUA ( Nie et al. , 2019 ) introduced DenseNet ’ s ( Huang et al. , 2016 ) densely connected structure , which summarizes the output of each layer and transmits it to the next layer . This structure forms a dense connection between the different layers of the network . The Stereo-DRNet ( Chabra et al. , 2019 ) introduced the vortex pooling structure ( Xie et al. , 2018 ) , which is a variant of the ASPP . In this structure , average pooling is carried out on the feature map before the dilated convolution , and the size of the pooling kernel is the corresponding dilation rate . Zhu et al . proposed the CFPNet ( Zhu et al. , 2019 ) and designed a cross form spatial pyramid pooling ( CFSPP ) module , which consist of two branches : one branch is the SPP structure , and the other branch is the ASPP structure . The feature maps obtained from two branches are concatenated to obtain the feature information of each scale . In the feature extraction network , almost all the existing stereo matching networks take the ResNet50 structure as their backbones . In this paper , we proposed a shallow feature extraction network with fewer parameters but a larger receptive field , whose matching accuracy exceeds all the above algorithms . 3 ARCHITECTURE . We propose a shallow feature extraction network with a large receptive field – SWNet – which consists of three parts : the primary feature extraction module ( PFE ) , the atrous spatial pyramid pooling ( ASPP ) module and the feature fusion module ( FFM ) . The network architecture is shown in Figure 1 . The detailed parameters of the feature extraction network structure that is designed in this paper are shown in Appendix B . The primary feature extraction network consists of three convolution layers with a kernel size of 3 * 3 , each of which is followed by a batch normalization layer ( Ioffe & Szegedy , 2015 ) and a ReLU layer . Only the stride of the first convolution layer is 2 to reduce the size of images . The other layers strides are set to 1 to retain more spatial structure information . Because of the shallow network structure , the size of the receptive field is limited . Therefore , inspired by Deeplab v2 ( Chen et al. , 2018 ) , the ASPP module is added to the PFE module . In this module , dilated convolution layers with different dilation rate ( e.g . 2 , 4 , 6 , and 8 ) form four parallel branches . The four branches have receptive fields with different scales , which can complement each other . The outputs of four branches are added to obtain the feature maps containing multiscale information . Unlike the processing method of directly summing the feature maps with multiscale receptive fields , this paper adopts a feature fusion module to integrate the feature information of different scales . First , the feature maps that are obtained from each branch are concatenated to form a feature map group . Since the importance of the information that is contained in each feature map is different , inspired by SENet ( Jie et al. , 2017 ) , this paper gives each feature map a specific weight . The feature fusion module is illustrated in Figure 1 . The feature map group is converted into a 1D feature vector by global average pooling , a bottleneck structure is used to limit the number of parameters , and the weight of each channel is obtained by using a sigmoid function . The bottleneck structure is composed of two 1 * 1 convolution layers and a ReLU activation layer . The first convolution layer compresses the number of channels by r times . After activation using the ReLU function , the number of channels is recovered by the second convolution layer . The weighted feature map group is obtained by multiplying the weight coefficient with the corresponding feature map . Then , the feature maps that are obtained by the PFE module are concatenated with the weighted feature map group through the skip connection , and the number of channels is compressed to 32 using two 3 * 3 convolution layers to obtain the final fusion feature maps .

Summary: The paper at hand argues that shallow feature extraction networks should be favored for the computer vision task of stereo matching, rather than the commonly used deep ResNet backbones. To that end, a model is proposed consisting of three convolutional feature extraction layers only. As this makes it impossible to capture global context, dilated convolutions with varying dilations are applied to the extracted features in parallel and concatenated. Finally, a fusion module implemented via channel attention is applied.