Depth estimation for advancing intelligent transport systems based on self‐improving pyramid stereo network

IET Intelligent Transport SystemsVolume 14, Issue 5 p. 338-345 Special Issue: AI Applications to Intelligent Vehicles for Advancing Intelligent Transport SystemFree Access Depth estimation for advancing intelligent transport systems based on self-improving pyramid stereo network Yanling Tian, Yanling Tian CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China Graduate School of Information, Production and Systems, Waseda University, JapanSearch for more papers by this authorYubo Du, Yubo Du CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of ChinaSearch for more papers by this authorQieshi Zhang, Corresponding Author Qieshi Zhang qs.zhang@siat.ac.cn CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China The Chinese University of Hong Kong, Hong Kong, People's Republic of ChinaSearch for more papers by this authorJun Cheng, Jun Cheng CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China The Chinese University of Hong Kong, Hong Kong, People's Republic of ChinaSearch for more papers by this authorZhuo Yang, Zhuo Yang School of Computers, Guangdong University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this author Yanling Tian, Yanling Tian CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China Graduate School of Information, Production and Systems, Waseda University, JapanSearch for more papers by this authorYubo Du, Yubo Du CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of ChinaSearch for more papers by this authorQieshi Zhang, Corresponding Author Qieshi Zhang qs.zhang@siat.ac.cn CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China The Chinese University of Hong Kong, Hong Kong, People's Republic of ChinaSearch for more papers by this authorJun Cheng, Jun Cheng CAS Key Laboratory of Human-Machine Intelligence-Synergy Systems, Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, People's Republic of China The Chinese University of Hong Kong, Hong Kong, People's Republic of ChinaSearch for more papers by this authorZhuo Yang, Zhuo Yang School of Computers, Guangdong University of Technology, Guangzhou, People's Republic of ChinaSearch for more papers by this author First published: 31 March 2020 https://doi.org/10.1049/iet-its.2019.0462Citations: 6AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat Abstract In autonomous driving, stereo vision-based depth estimation technology can help to estimate the distance of obstacles accurately, which is crucial for correctly planning the path of the vehicle. Recent work has formulated the stereo depth estimation problem into a deep learning model with convolutional neural networks. However, these methods need a lot of post-processing and do not have strong adaptive capabilities to ill-posed regions or new scenes. In addition, due to the difficulty of the labelling the ground truth depth for real circumstance, training data for the system is limited. To overcome the above problems, the authors came up with self-improving pyramid stereo network, which can not only get a direct regression disparity without complicated post-processing but also be robust in ill-posed area. Moreover, by online learning, the proposed model can not only address the data limitation problem but also save the time spent on training and hardware resources in practice. At the same time, the proposed model has a self-improving ability to new scenes, which can quickly adjust the model according to the test data in time and improve the accuracy of prediction. Experiments on Scene Flow and KITTI data set demonstrate the effectiveness of the proposed network. 1 Introduction Depth estimation is an emerging focus in autonomous driving (AD) [1-4]. For any vehicle, depth information is important for environmental perception to determine the position and the surrounding obstacles in the environment. In the same time, depth estimation from stereo images is a hot research field. The camera is one of the important sensors in AD. The primary task in the AD is collecting information, especially depth information, in obstacle detection and object recognition [5-7]. In the past, depth information was always obtained by depth camera or light detection and ranging (LiDAR). However, in addition to the huge calculations required by sensors, high-quality LiDAR is still expensive and depth cameras often produce hard-to-remove noise in outdoor environments. Recently, deep learning-based depth estimation comes into sight using the generation model [8-10], which brings a new solution for environmental perception. More and more monocular camera-based depth estimation algorithms are developed. However, compared to stereo vision algorithm, a monocular camera algorithm has inherent defects. First, vision field information cannot be enriched. Second, since the monocular camera-based depth estimation only contains images of one frame, it is extremely difficult to accurately estimate the depth information, which cannot meet the actual needs. Stereo vision-based depth estimation mimics the two eyes structural characteristics of the creature, using the known geometric information between the two cameras for depth estimation. This problem is modelled as matching corresponding points of a pair of rectified pictures, and this type of relation will be described as a disparity map. As shown in Fig. 1, the blue line presents the image planes. If a point in the left image matches the point found at in right image, the disparity of point in the disparity map is . Then we can calculate the depth of this point by , where f is the camera focal length and b is the distance between two cameras' centres. Fig. 1Open in figure viewerPowerPoint Binocular camera imaging principle. Y-axis is perpendicular to the page To solve above problem, deep learning-based stereo matching method was first proposed by Zbontar and LeCun, in 2016 [11], which achieved an extraordinary result on KITTI [12, 13], and many other related works emerged [14, 15]. However, these methods [11, 16] needed to not only classify the image patches into good matches and bad matches but also had a time-consuming post-processing. To overcome these shortcomings, researchers found that the cues such as perspective, scaling relative to the known size of familiar objects, appearance in the form of lighting and shading and occlusion could be exploited just as human eyes did with convolutional neural network (CNN) [17]. Then, by learning corresponding relations from the similarities of the cues, which were subtracted from the images, the pixel-level correspondence between pairs of rectified stereo images could be predicted directly. To increase the reliability of correspondence, people also proposed a geometry and context network (GC-Net) which created an end-to-end disparity regression structure [18]. In addition, according to [19], a novel pyramid stereo match network (PSMNet) was proposed. Although the CNN-based approaches [20, 21] improved performance of stereo depth estimation on challenging public benchmarks, current CNNs are usually constrained by the data set. They cannot produce satisfying results for real-life photographs due to the limitation of the data set. First, the difficulty of getting depth data from LiDAR limits the number and variety of the data set, so the model trained from them do not have strong adaptive ability to different scenes. Second, the data distortion of radar also affects the validity of the ground truth depth data. To get rid of this limitation, several unsupervised methods [17, 22] were proposed to improve the adaption ability of models. Instead of only learning from the ground truth data, they also learned from reconstruction loss, such as the loss of reconstructing left input image through predicted right disparity. To overcome the above shortcomings and to utilise above advantage, based on GC-Net and PSMNet, we propose the self-improving pyramid stereo network (SIPSNet) which is trained in an end-to-end manner without any post-processing. In addition, the stereo disparity maps, which are generated through the spatial pyramid pooling (SPP) module, cross cost volume and hierarchical disparity regression can produce a stereo reconstruction loss so that our model can work without ground truth data. Our work makes three contributions: (i) Construct reconstruction loss, applied in a multi-layer regression outputs architecture, which generates stereo outputs so that it can work without ground-truth data and have the strong ability for adapting to new scenes. (ii) Propose SIPSNet framework, which combines the end-to-end structure and the SPP module to avoid time-consuming post-processing and to capture the information even in ill-posed regions. (iii) Conduct experiments to show that our unsupervised method can be trained in a quicker and more accurate manner compared with the-state-of-art methods, and get comparable results with PSMNet, a strong-supervised method. The rest parts are structured as follows: in Section 2, we introduce related work including monocular camera-based depth estimation and stereo vision-based depth estimation algorithm. Then Section 3 refers to the details of our proposed method. Experiments are in Section 4, we will make some comparisons with state-of-the-art methods. Lastly, the conclusion is summarised in Section 5. 2 Related work Depth estimation algorithms based on depth learning are generally divided into monocular camera-based depth estimation and stereo vision-based depth estimation algorithms. We will introduce related work from these two aspects: 2.1 Monocular camera-based depth estimation Monocular depth estimation has great significance in the field of AD [8-10], but it needs to consider local information, global information and single-purpose clues like line angles, perspectives, object shapes and environmental influences. David Eigen et al. [8] proposed two deep networks: one of them makes a coarse global prediction based on the entire image and another one refines its prediction locally. In addition, they applied scale-invariant error to measure depth relations rather than scale, which makes model scale to the appropriate size by the post-processing. Laina et al. [9] proposed a fully convolutional residual network to estimate deeper depth prediction, which efficiently learns the feature map up-sampling and does not rely on post-processing techniques within the network. The above methods used a single view image data as an input, although they used implicit geometric constraints in the training process which can reduce the dependence on the data, these methods still lack explicit geometric constraints during the test which leads to suboptimal results. Then, Luo et al. [23] proposed single view stereo matching, which explicitly imposed geometrical constraints during inference and greatly alleviated the demand on labelled depth data. 2.2 Stereo vision-based depth estimation Traditional monocular camera-based depth estimation does not work well on. This makes more researchers focus on stereo vision [18, 19, 24], which can obtain the depth information between the images according to the change of the two picture angles. Kendall et al. [18] proposed a GC-Net. In this model, the unary features, which learned through a number of two-dimensional (2D) convolutional operations, are concatenated at each disparity level into a 4D cost volume. Then they applied a regularisation function to take into account context in this volume and refine disparity estimate. Chang et al. [19] proposed a PSMNet to exploit global context information for stereo matching without any post-processing. In addition, an SPP module was added to the GC-Net and PSMNet was proposed. With the multi-scale pooling operations, the network could get a larger receptive field and exploit the information even in the ill-posed area. Although the above methods achieved great performance, they cannot meet actual application due to the adaptive ability to different scenes in the limited data set. Zhang et al. [24] proposed an ActiveStereoNet and introduced a new reconstruction loss, which is used to solve the problem of the lack of ground truth data in a fully self-supervised way. 3 Our method Our SIPSNet consists of four parts: (i) SPP module enlarges the receptive field of the network, (ii) cost volume transforms the aggregation of 2D features into 4D volume and enables the network to be trained in an end-to-end manner, (iii) multi-layer regression outputs improve the accuracy, (iv) reconstruction loss of stereo output makes the model be able to work without ground truth data. The structure is shown in Fig. 2. Then we will introduce these four parts as follows, respectively. Fig. 2Open in figure viewerPowerPoint Structure of our model: the stereo inputs are first fed into SPP module to extract features then they are concatenated at each disparity level, and after a basic residual block they are finally put into a multi-level output structure. Part A is the cost volume module and details of two different kinds of features concatenating are shown in part A1 and part A2. When the features are scattered into each representation at each disparity level they become weaker features than put together at each disparity level. In part A, blue squares represent the right features and orange ones represent left features. Part B shows the structure of the multi-level output 3.1 Spatial pyramid pooling (SPP) module SPP module was designed to overcome the constraint in the size of CNN [25]. It was commonly applied in semantic segmentation problems with the ability to combining global and regional context information to attain a larger receptive field. SPP model is used to solve the stereo depth estimation problem and is proved that the combination of multi-size pooling feature extraction performed better than fixed-size pooling feature extracting on this problem [19]. Similar with [19], we first establish two basic residual blocks and two dilated convolution layers to pre-process features. Then they are followed by four fixed-size pooling blocks to further extract features, as shown in Fig. 3. Fig. 3Open in figure viewerPowerPoint Spatial pyramid pooling (SPP) module 3.2 Cost volume Inspired by [18], the learned features are used to compute the stereo matching cost by constructing a 4D cost volume constructed by exhausting disparity levels in a pre-fined range (shown in Fig. 2 : part A). There are usually two kinds of cost volume aggregation: Concatenating all the features in each disparity level (shown in Fig. 2 : part A1). Concatenating each feature in each disparity level (shown in Fig. 2 : part A2). In our work, we find that the first aggregation is superior to the second one, because putting the features together generates a stronger feature representation than scattering them. So the first one is applied in our SIPSNet. Effects of different cost volume aggregations are shown in Section 4.2. 3.3 Multi-layer regression outputs To exploit context information along the disparity dimensions, we use a repeated encoder and decoder architecture (shown in Fig. 2 : part B). It consists of three pairs of encoder–decoder module and each of them generates a 4D disparity volume , showing the normalised possibility of each pixel at each disparity. Instead of optimising only with the last output, parameters converge more quickly by optimising hierarchically. To get a succinct output, we use the following operation to generate a 2D disparity map: (1) where is the predicted disparity and is the normalised possibility. In the training part, the weighted sum of three outputs is calculated as the total loss function. In the testing part, the final disparity map is the last of three outputs. 3.4 Reconstruction loss Our model can work both in the supervised manner with ground truth data and unsupervised manner with a simple but efficient loss function. 3.4.1 Supervised loss In the supervised manner, the loss is defined as the error between predicted disparity and ground truth data. Because the Smooth Loss is more robust and less sensitive to outliers than Loss according to [19]. The supervised loss function is as following: (2) in which (3) where is the ground truth disparity, is the predicted disparity, N is the number of pixels with labelled disparity. In our work, the supervised loss is only computed on left images due to the limitation of the data set. 3.4.2 Unsupervised loss Owing to the stereo outputs, SIPSNet can be optimised by stereo reconstruction loss and work in an unsupervised way. In similar works [17, 26], they use many other complicated losses such as the loss of reconstructed disparity map, the disparity gradient loss, the loss of twice reconstructed image. In our work, these losses actually will bring down the accuracy and increase the computing time, so we only use the simplified but the most efficient loss. Inspired by [26], we combine the single scale structural similarity (SSIM) index measurement term and smooth loss as the unsupervised reconstruction error (4) where denote SSIM loss between image patches, denote smooth loss between image patches. SSIM loss (5) The SSIM here uses a block filter but not Gaussian. Smooth loss (6) we set . Reconstruct the opposite picture. In (5) and (6) (7) We use the bilinear image sampler from the spatial transformer network [27] to sample the reconstructed left image by computing the weighted sum of four nearest pixels around each pixel. As for the right image, we deal with it symmetrically as the right image. 4 Experiment In this section, we first conduct the ablation study to discuss the effects of cost volume aggregation. And then, SIPSNet is analysed on KITTI 2015 [13, 28] to prove that it can be put into use even without ground truth data or pre-trained model and the self-improving ability of SIPSNet. The self-improving ability of SIPSNet in different circumstances is tested on KITTI 2012 [12] and Scene Flow (https://lmb.informatik.unifreiburg.de/resources/datasets) data set. The performance of our SIPSNet is compared with other state-of-the-art methods on KITTI 2015 and Scene Flow benchmark finally. 4.1 Implementation details The proposed SIPSNet runs on Pytorch, with GPUs. For KITTI data set, it is a real-word data set with street views from a driving car. In KITTI 2015 data set, it contains 200 training stereo image pairs and 200 testing image pairs. Also, we divide the training set into160 image pairs for training and 40 image pairs for validation. In KITTI 2012 data set, there are 194 training stereo image pairs and 195 testing image pairs. In addition, we divide the training set into 160 image pairs for training and 34 image pairs for validation. Scene Flow data set is a synthetic stereo data set which contains 35,454 training and 4370 testing images of size 960 × 540. We train and test our model on data set divided by Scene Flow. During optimisation, due to the limitation of GPU, we set , the picture size as 256 × 512 in the training, 368 × 1232 in the testing and . Our model was optimised with momentum optimiser (set in SGD optimiser). Besides, stereo branches share the same parameters both in the training and testing process. The input images in training are randomly cropped from a pair of normalised colour stereo images and the disparities in ground truth maps range from 0 to 256. 4.2 Ablation study We conducted experiments with different settings to evaluate SIPSNet, including usage of SPP, cost volume aggregation ways and multi-layer regression outputs architectures. As listed in Table 1, we can find that SPP works better than the basic network. For cost volume aggregation, whether with or without multi-layer regression outputs architecture, the result with cost volume aggregation 1 (Fig. 2 : part A1) works better than cost volume aggregation 2 (Fig. 2 : part A2). When combined with SPP and cost volume aggregation 1 (Fig. 2 : part A1), the multi-layer regression outputs architecture significantly works better, with 1.71% error rate on the KITTI 2015 validation set and best average endpoint error (EPE) of 0.81 pixel on the Scene Flow test set. Table 1. Evaluation of SIPSNet with different settings. We computed the percentage of three-pixel-error on the KITTI 2015 validation set and EPE on the Scene Flow test set Network setting KITTI 2015 Scene Flow SPP Cost volume aggregation Multi-layer regression outputs Val Err, % EPE Part A1 Part A2 2.46 1.58 2.13 1.26 1.97 1.07 2.02 1.12 1.71 0.81 1.87 1.09 EPE, endpoint error. Bold values indicate the best result. 4.3 Self-improving ability In this section, we first prove our proposed method SIPSNet can be trained by unsupervised and supervised way, and then self-improving ability of SIPSNet is tested on original data set and new scenes, respectively. To prove that SIPSNet can work without ground truth data or pre-trained model, we trained SIPSNet without any pre-trained model in two ways: with ground truth data and without ground truth data. As shown in Fig. 4, we can see that the unsupervised loss on test data converges at 8.017% and the supervised loss converges at 3.726%. It shows that, without ground truth data, SIPSNet can reach a reasonable result even compared with the results from supervised learning. Fig. 4Open in figure viewerPowerPoint 3-pixel loss on test data Experiments about self-improving ability of SIPSNet are conducted First, we test self-improving ability on the original data set, based on pre-trained model [19], compared with a strong-supervised method PSMNet. For KITTI data set, the PSMNet takes 30 training epochs to fine-turn and SIPSNet takes 30 online improving iterations to test directly with unsupervised manner on the test set. For Scene Flow data set, the PSMNet takes five training epochs and SIPSNet takes 30 online improving iterations. As shown in Table 2, the result shows that SIPSNet can reach a reasonable result and spend less time, after online improving but without training. Table 2. Self-improving ability on the percentage of pixels with errors of more than three pixels and runtime of SIPSNet compared with PSMNet method. 3-pixel error: the percentage of pixels with errors of more than three pixels Data set Method KITTI 2015 KITTI 2012 Scene Flow 3-pixel error, % SIPSNet 0.892 0.921 4.912 PSMNet [19] 0.877 0.924 4.800 time(h*GPU) SIPSNet 0.29 0.29 2.12 PSMNet [19] 0.84 0.85 5.32 Bold values indicate the best result. Take KITTI 2012 data set as an example, qualitative comparison is shown in Fig. 5. Fig. 5Open in figure viewerPowerPoint Qualitative comparison about the self-improving ability of SIPSNet on the original data set Second, we test the self-improving ability on new scenes. If the model M is trained on data set A then data set B is a new scene to M. So we test the pre-trained model from KITTI 2015 on KITTI 2012 and Scene Flow data set. Without any further supervised training, we take only 30 online improving iterations. As shown in Table 3, the significant decrease of 3-pixel error (the percentage of pixels with errors of more than three pixels) after online improving shows the effectiveness of SIPS' self-improving ability to new scenes. In addition, some images that do not belong to the training data set are placed to prove the self-improving response to the new scenes as shown in Fig. 6. Table 3. Comparing the adaptive ability to new scenes on different data sets. The following pre-trained model is trained on KITTI 2015 and for the model, images form KITTI 2012 and Scene Flow are the new scenes. 3-pixel error: the percentage of pixels with errors of more than three pixels Architecture KITTI 2012 Scene Flow 3-pixel error, % SIPSNet 3.145 10.372 — pre-trained model 8.702 18.240 Fig. 6Open in figure viewerPowerPoint Qualitative comparison about the self-improving ability of SIPSNet on new scenes Therefore, our proposed method SIPSNet, through online improving iterations, can save the time spent on training and hardware resources in practice. In addition, it can quickly adjust the model according to the online test data, so that the model can reflect the changes in time and improve the accuracy of prediction. it is more suitable and economical for real applications. 4.4 Scene Flow data set We compare the EPE performance of the proposed method SIPSNet with other state-of-the-art methods which have published the codes, including PSMNet [19], GC-Net [18], PDSNet [29], GwcNet-g [30] and GA-Net [31] on the Scene Flow test set as listed in Table 4. Although GA-Net outperform our proposed method SIPSNet on KITTI 2015 data set (which background error: 1.34%, foreground error: 3.11%, error in all ground truth: 1.63% and runtime:1.8 s showed on KITTI 2015 leaderboard), SIPSNet outperforms GA-Net in terms of EPE, 1-pixel threshold error rate and times on Scene Flow test set. Table 4. Performance comparison with Scene Flow test set Methods EPE Error rates, % Time, s PDSNet [29] 1.42 13.2 2.9 GC-Net [18] 1.80 15.6 4.4 PSMNet [19] 1.09 12.1 2.1 GwcNet-g [30] 0.99 10.8 1.7 GA-Net [31] 1.07 11.9 0.6 SIPSNet 0.81 8.3 0.5 EPE: endpoint error. Bold values indicate the best result. 4.5 KITTI 2012 and 2015 data sets We analyse the experiment results from the quantitative evaluation and qualitative evaluation: 4.5.1 Quantitative evaluation In Table 5, we compare the performance of our model with the most related methods and other state-of-the-art methods on KITTI 2015 test set average ( : background error; : foreground error; : error in all ground truth), including PDSNet [29], GC-Net [18], PSMNet [19], GwcNet-g [30], MCUA [32] and EMCUA [32]. The SIPSNet is trained for 400 epochs (in Fig. 2 SIPSNet converges after 300 epochs) on pre-trained model [19], and then takes 10 online improving iterations (we test the best online improving iteration on the testing data set divided by ourselves on KITTI 2015 and the best online improving epoch average is 10) on test data. The foreground error and all ground truth error of our SIPSNet transcend other methods. Among these methods on all ground truth error, PSMNet takes 1000 training epochs, while our method saves more time. Table 5. Performance of different methods on KITTI 2015 data set Method , % , % , % PDSNet [29] 2.29 4.05 2.58 GC-Net [18] 2.21 6.16 2.87 PSMNet [19] 1.86 4.62 2.32 GwcNet-g [30] 1.74 3.93 2.11 MCUA [32] 1.69 4.38 2.14 EMCUA [32] 1.66 4.27 2.09 SIPSNet 1.74 3.76 2.01 : background error; : foreground error; : error in all ground truth. Bold values indicate the best result. We choose three test images for comparison from KITTI 2015, as shown in Table 6 and the corresponding qualitative evaluation is illustrated in Section 4.5.2. Table 6. Performance of different methods on KITTI 2015 data set for several test images Method Image 1, % Image 2, % Image 3, % PDSNet [29] 1.13 3.60 1.54 0.79 3.97 1.42 3.39 2.29 3.75 GC-Net [18] 1.96 1.73 1.92 0.83 3.01 1.25 5.29 3.41 5.12 PSMNet [19] 1.56 0.78 1.43 0.58 3.15 1.08 3.67 1.16 3.45 GwcNet-g [30] 1.52 0.22 1.29 0.41 2.57 0.83 3.18 1.78 3.05 MCUA [32] 1.76 0.73 1.59 0.38 2.99 0.89 2.78 1.67 2.68 SIPSNet 0.81 0.43 0.79 0.51 3.48 1.08 2.45 1.54 2.44 : background error; : foreground error; : error in all ground truth. Bold values indicate the best result. 4.5.2 Qualitative evaluation As shown in Fig. 7, there are some examples of the disparity maps and the corresponding error maps estimated by the proposed SIPSNet, PDSNet [29], GC-Net [18], PSMNet [19], GwcNet-g [30] and MCUA [32] on KITTI 2015 data set. We can see that our proposed method SIPSNet yields more robust results, particularly in ill-posed regions. Compared to the other three methods, SIPSNet more accurately predicts tree and road sign. In Fig. 5, compared with PSMNet [19] on KITTI 2012 data set, SIPSNet can better recover detailed scene structures. such as fence marked out with black dashed bounding boxes at the first row. In addition, SIPSNet still needs to be improved due to certain large textureless regions such as the bottom row but is still better than PSMNet [19]. Fig. 7Open in figure viewerPowerPoint Results of disparity estimation for KITTI 2015 test images. The top panel shows the input image. For each input image, the disparity maps (top row) obtained by (a) PDSNet [29], (b) GC-Net [18], (c) PSMNet [19], (d) GwcNet-g [30], (e) MCUA [32], (f) SIPSNet are illustrated in the bottom row above their error maps 5 Conclusion In our work, we proposed an end-to-end SIPSNet which has a strong self-improving and adaptive ability to new scenes. When there is no ground truth data or pre-trained model, working with only online learning can achieve an excellent result. Besides, when applied to new scenes, our model can adapt to new scenes with only a few online learning iterations and reach a reasonable performance. In addition, our model can save hardware resources in practice and quickly adjust the model according to the online test data, which is more suitable and economical for real applications. 6 Acknowledgments This work was supported in part by the National Key Research and Development Program of China (No. 2018YFB1308000), in part by the National Natural Science Funds of China (No. U1913202, U1813205, U1713213, 61772508, 61907009), the Natural Science Foundation of Guangdong Province (No. 2018A030313802), the Shenzhen Technology Project (No. JCYJ20180507182610734, JCYJ20170413152535587), and in part by the CAS Key Technology Talent Program. 7 References 1Chen C. Seff A. Kornhauser A. et al.: ' Deepdriving: learning affordance for direct perception in autonomous driving'. IEEE Int. Conf. on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 2722– 2730 2Smolyanskiy N. Kamenev A. Birchfield S.: ' On the importance of stereo for accurate depth estimation: an efficient semi-supervised deep neural network approach'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 4040– 4048 3Achtelik M. Bachrach A. He R. et al.: ' Stereo vision and laser odometry for autonomous helicopters in GPS-denied indoor environments'. Int. Society for Optical Engineering, Orlando, FL, USA, 2009, vol. 7332, (733219), pp. 1– 10 4Mayer N. Ilg E. Häusser P. et al.: ' A large dataset to train convolutional networks for disparity, optical flow, and scene flow estimation'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 4040– 4048 5Chen X. Kundu K. Zhu Y. et al.: ' 3D object proposals for accurate object class detection'. Int. Conf. on Neural Information Processing Systems (NIPS), Montréal, Canada, 2015, pp. 424– 432 6Girshick R.B. Donahue J. Darrell T. et al.: ' Rich feature hierarchies for accurate object detection and semantic segmentation'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Columbus, OH, USA, 2014, pp. 580– 587 7Güney F. Geiger A.: ' Displets: resolving stereo ambiguities using object knowledge'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 4165– 4175 8Eigen D. Puhrsch C. Fergus R.: ' Depth map prediction from a single image using a multi-scale deep network'. Int. Conf. on Neural Information Processing Systems (NIPS), Montréal, Canada, 2014, pp. 2366– 2374 9Laina I. Rupprecht C. Belagiannis V. et al.: ' Deeper depth prediction with fully convolutional residual networks'. Int. Conf. on 3D Vision, Stanford, CA, USA, 2016, pp. 239– 248 10Eigen D. Fergus R.: ' Predicting depth, surface normals and semantic labels with a common multi-scale convolutional architecture'. IEEE Int. Conf. on Computer Vision (ICCV), Santiago, Chile, 2015, pp. 2650– 2658 11Zbontar J. LeCun Y.: 'Stereo matching by training a convolutional neural network to compare image patches', J. Mach. Learn. Res. (JMLR), 2016, 17, (1), pp. 2287– 2318 12Geiger A. Lenz P. Urtasun R.: ' Are we ready for autonomous driving? The KITTI vision benchmark suite'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Providence, RI, USA, 2012, pp. 3354– 3361 13Menze M. Heipke C. Geiger A.: ' Joint 3D estimation of vehicles and scene flow'. Photogrammetry, Remote Sensing and Spatial Information Sciences, La Grande Motte, France, 2015 vol II-3/W5, pp. 427– 434 14Luo W. Schwing A.G. Urtasun R.: ' Efficient deep learning for stereo matching'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, 2016, pp. 5695– 5703 15Pang J. Sun W. Ren J. S. J. et al.: ' Cascade residual learning: A two-stage convolutional neural network for stereo matching'. IEEE Int. Conf. on Computer Vision Workshop (ICCVW), Venice, Italy, 2017, pp. 887– 895 16Zbontar J. LeCun Y.: ' Computing the stereo matching cost with a convolutional neural network'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Boston, MA, USA, 2015, pp. 1592– 1599 17Godard C. Aodha O.M. Brostow G.J.: ' Unsupervised monocular depth estimation with left-right consistency'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 6602– 6611 18Kendall A. Martirosyan H. Dasgupta S. et al.: ' End-to-end learning of geometry and context for deep stereo regression'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 5410– 5418 19Chang J.-R. Chen Y.-S.: ' Pyramid stereo matching network'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Salt Lake City, UT, USA, 2018, pp. 5410– 5418 20Liu F. Shen C. Lin G. et al.: 'Learning depth from single monocular images using deep convolutional neural fields', IEEE Trans. Pattern Anal. Mach. Intell. (TPAMI), 2015, 38, (10), pp. 2024– 2039 21Hirschmüller H. Scharstein D.: ' Evaluation of cost functions for stereo matching'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Minneapolis, MN, USA, 2007, pp. 1– 8 22Garg R. Vijay Kumar B.G. Carneiro G. et al.: ' Unsupervised CNN for single view depth estimation: geometry to the rescue'. European Conf. on Computer Vision (ECCV), Amsterdam, The Netherlands, 2016, pp. 740– 756 23Luo Y. Ren J. Lin M. et al.: ' Single view stereo matching'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Lake City, UT, USA, 2018, pp. 155– 163 24Zhang Y. Khamis S. Rhemann C. et al.: ' Activestereonet: End-to-end self-supervised learning for active stereo systems'. European Conf. on Computer Vision (ECCV), Munich, Germany, 2018, pp. 802– 819 25Zhao H. Shi J. Qi X. et al.: ' Pyramid scene parsing network'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Honolulu, HI, USA, 2017, pp. 6230– 6239 26Zhou C. Zhang H. Shen X. et al.: ' Unsupervised learning of stereo matching'. IEEE Int. Conf. on Computer Vision (ICCV), Venice, Italy, 2017, pp. 1576– 1584 27Jaderberg M. Simonyan K. Zisserman A. et al.: ' Spatial transformer networks'. Advances in Neural Information Processing Systems (NIPS), Montréal, Canada, 2015, pp. 2017– 2025 28Menze M. Heipke C. Geiger A.: 'Object scene flow', ISPRS J. Photogramm. Remote Sens. (JPRS), 2018, 140, pp. 60– 76 29Tulyakov S. Ivanov A. François F.: ' Practical deep stereo (PDS): toward applications-friendly deep stereo matching'. Advances in Neural Information Processing Systems (NIPS), Montréal, Canada, 2018, pp. 5875– 5885 30Guo X. Yang K. Yang W. et al.: ' Group-wise correlation stereo network'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3273– 3282 31Zhang F. Prisacariu V. Yang R. et al.: ' GA-Net: guided aggregation net for end-to-end stereo matching'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 185– 194 32Nie G.-Y. Cheng M.-M. Liu Y. et al.: ' Multi-level context ultra-aggregation for stereo matching'. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), Long Beach, CA, USA, 2019, pp. 3283– 3291 Citing Literature Volume14, Issue5Special Issue: AI Applications to Intelligent Vehicles for Advancing Intelligent Transport SystemsMay 2020Pages 338-345 FiguresReferencesRelatedInformation