SBEVNet: End-to-End Deep Stereo Layout Estimation

1 INTRODUCTION . Layout estimation is an extremely important task for navigation and planning in numerous robotics applications such as autonomous driving cars . The bird ’ s eye view ( BEV ) layout is a semantic occupancy map containing per pixel class information , e.g . road , sidewalk , cars , vegetation , etc . The BEV semantic map is important for planning the path of the robot in order to prevent it from hitting objects and going to impassable locations . In order to generate a BEV layout , we need 3D information about the scene . Sensors such as LiDAR ( Light Detection And Ranging ) can provide accurate point clouds . The biggest limitations of LiDAR are high cost , sparse resolution , and low scan-rates . Also , as an active sensor LiDAR is more power hungry , more susceptible to interference from other radiation sources , and can affect the scene . Cameras on the other hand , are much cheaper , passive , and capture much more information at a higher frame-rate . However , it is both hard and computationally expensive to get accurate depth and point clouds from cameras . The classic approach for stereo layout estimation contains two steps . The first step is to generate a BEV feature map by an orthographic projection of the point cloud generated using stereo images . The second step is bird ’ s eye view semantic segmentation using the projected point cloud from the first step . This approach is limited by the estimated point cloud accuracy because the error in it will propagate to the layout estimation step . In this paper , we show that explicit depth estimation is actually neither sufficient nor necessary for good layout estimation . Estimating accurate depth is not sufficient because many areas in the 3D space can be occluded partially , e.g . behind a tree trunk . 1The code and the synthesized dataset will be made public upon the acceptance of this paper . However , these areas can be estimated by combining spatial reasoning and geometric knowledge in bird ’ s eye view representation . Explicitly estimating accurate depth is also not necessary because layout estimation can be done without estimating the point cloud . Point cloud coordinate accuracy is limited by the 3D to 2D BEV projection and rasterization . For these reasons , having an effective bird ’ s eye view representation is very important . SBEVNet is built upon recent deep stereo matching paradigm . These deep learning based methods have shown tremendous success in stereo disparity/depth estimation . Most of these models ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) generate a 3-dimensional disparity feature volume by concatenating the left and right images shifted at different disparities , which is used to make a cost volume containing stereo matching costs for each disparity value . Given a location in the image and the disparity , we can get the position of the corresponding 3D point in the world space . Hence , every point in the feature volume and cost volume corresponds to a 3D location in the world space . The innovation in our approach comes from the observation : it is possible to directly use the feature volume for layout estimation , rather than a two step process , which uses the point cloud generated by the network . We propose SBEVNet , an end-to-end neural architecture that takes a pair of stereo images and outputs the bird ’ s eye view scene layout . We first project the disparity feature volume to the BEV view , creating a 2D representation from the 3D volume . We then warp it by mapping different disparities and the image coordinates to the bird ’ s eye view space . In order to overcome the loss of fine grained information imposed by our choice of the stereo BEV feature map , we concatenate a projection of the original images and deep features to this feature map . We generate these projected features by applying inverse perspective mapping ( IPM ) ( Mallot et al. , 1991 ) to the input image and its features , choosing the ground as the target plane We feed this representation to a U-Net in order to estimate the BEV semantic map of the scene . In order to perform inverse perspective mapping , we require information about the ground in the 3D world space . Hence we also consider the scenario where we perform IPM during the training time and not the inference time . Here , during the training time , we use cross modal distillation to transfer knowledge from IPM features to the stereo features . SBEVNet is the first approach to use an end-to-end neural architecture for stereo layout estimation . We show that SBEVNet achieves better performance than existing approaches . SBEVNet outperforms all the baseline algorithms on KITTI ( Geiger et al. , 2013 ) dataset and a synthetically generated dataset extracted from the CARLA simulator ( Dosovitskiy et al. , 2017 ) . In summary , our contributions are the following : 1 . We propose SBEVNet , an end-to-end neural architecture for layout estimation from a stereo pair of images . 2 . We learn a novel representation for BEV layout estimation by fusing projected stereo feature volume and fine grained inverse perspective mapping features . 3 . We evaluate SBEVNet and demonstrate state-of-the-art performance over other methods by a large margin on two datasets – KITTI dataset and our synthetically generated dataset using the CARLA simulator . 2 RELATED WORK . To the best of our knowledge , there is no published research paper for estimating layout given a pair of stereo images . However , there are several works tackling layout estimation using a single image or doing object detection using stereo images . In this section , we review the most closely related approaches . Monocular Layout Estimation MonoLayout ( Mani et al. , 2020 ) uses an encoder-decoder model to estimate the bird ’ s eye view layout using a monocular input image . They also leverage adversarial training to produce sharper estimates . MonoOccupancy ( Lu et al. , 2019 ) uses a variational encoderdecoder network to estimate the layout . Both MonoLayout and MonoOccupancy do not use any camera geometry priors to perform the task . Schulter et al . ( 2018 ) uses depth estimation to project the image semantics to bird ’ s eye view . They also use Open Street Maps data to refine the BEV images via adversarial learning . Wang et al . ( 2019c ) uses ( Schulter et al. , 2018 ) to estimate the parameters of the road such as lanes , sidewalks , etc . Monocular methods learn strong prior , which does not generalize well when there is a significant domain shift . Stereo methods learn weak-prior plus geometric relationship , which can generalize better . Deep Stereo Matching Several methods like ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) extract the features of the stereo images and generate a 3D disparity feature volume for disparity/depth estimation . They use a 3D CNN on the feature volume to get cost volume to perform stereo matching . PSMNet ( Chang & Chen , 2018 ) uses a spatial pyramid pooling module and a stacked hourglass network to further improve the performance . High-res-stereo ( Yang et al. , 2019 ) uses a hierarchical model , creating cost volumes at multiple resolutions , performing the matching incrementally from over a coarse to fine hierarchy . Bird ’ s Eye View Object Detection Several approaches like Yang et al . ( 2018 ) ; Shi et al . ( 2019 ) use LiDAR to perform 3D object detection . Pseudo-lidar ( Wang et al. , 2019a ) and pseudo-LiDAR++ ( You et al. , 2019 ) use stereo input to first generate a 3D point cloud and then use a 3D object detection network ( Ku et al. , 2018 ; Yang et al. , 2018 ; Shi et al. , 2019 ) on top . BirdGAN ( Srivastava et al. , 2019 ) maps the input image to bird ’ s eye view using adversarial learning . The closest work to our approach is DSGN ( Chen et al. , 2020 ) which constructs a depth feature volume and map it to the 3D space which is then projected to bird ’ s eye view to perform object detection . The task of object detection is of sparse prediction , whereas layout estimation is of dense fine granularity prediction . Hence we introduced IPM to fuse low level detail with the stereo information to improve the performance of layout estimation . 3 OUR METHOD . This section describes the detailed architecture of our proposed framework . SBEVNet is built upon recent deep stereo matching paradigms and follows the rules of multi-view camera geometry . An overview of the SBEVNet is summarized in Figure 1 . 3.1 PROBLEM FORMULATION . In this paper , we address the problem of layout estimation from a pair of stereo images . Formally , given a reference camera image IR and a target camera image IT both of size H×W×3 , the camera intrinsics K , and the baseline length Tb , we aim to estimate the bird ’ s eye view layout of the scene . In particular , we estimate the BEV semantic map of size Nx × Ny × NC within the rectangular range of interest area ( xmin , xmax , ymin , ymax ) in front of the camera . Here H is image height , W is image width , and 3 indicates RGB channels . Nx and Ny are the number of horizontal cells and vertical cells respectively in bird ’ s eye view . NC is the number of semantic classes . This BEV semantic map contains the probability distribution among all semantic classes at each cell of the layout . We assume that the input images are rectified . 3.2 FEATURE EXTRACTION . The first step for SBEVNet is to extract features FR and FT of size H ′ ×W ′ × C for the reference image and the target image respectively . This is done by passing IR and IT through a convolutional encoder with shared weights . This produces multi-channel down-sized feature representations which are next used for building disparity feature volumes . 3.3 DISPARITY FEATURE VOLUME GENERATION . Similar to ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) we form a disparity feature volume V by concatenating the features FR and F dT , where F d T is FT shifted horizontally by a disparity of d pixel , resulting in a 3D volume of size H ′ × W ′ × D × 2C . We then pass the feature volume through a series of 3D convolution layers with skip connections to learn higher level features . This feature volume at each d ∈ { 0 , 1 , · · · , D − 1 } contains a representation of the 3D world at the depth corresponding to the disparity d. Rather than using this feature volume to do disparity estimation , we project and warp it to form a bird ’ s eye view representation in the next step .

Summary: The paper proposed to estimate the semantic layout in the bird eye's view from a pair of stereo images. The main novelty/contribution lies in how to organize and exploit the information from the stereo images. The proposed framework builds upon inverse perspective mapping, and projected stereo feature volume. The performance was evaluated on the KITTI and CARLA datasets. Given a pair of stereo images, there are various options to exploit the image information, where this paper provides a framework by exploiting the stereo information in the bird eye's view.