SBEVNet: End-to-End Deep Stereo Layout Estimation

1 INTRODUCTION . Layout estimation is an extremely important task for navigation and planning in numerous robotics applications such as autonomous driving cars . The bird ’ s eye view ( BEV ) layout is a semantic occupancy map containing per pixel class information , e.g . road , sidewalk , cars , vegetation , etc . The BEV semantic map is important for planning the path of the robot in order to prevent it from hitting objects and going to impassable locations . In order to generate a BEV layout , we need 3D information about the scene . Sensors such as LiDAR ( Light Detection And Ranging ) can provide accurate point clouds . The biggest limitations of LiDAR are high cost , sparse resolution , and low scan-rates . Also , as an active sensor LiDAR is more power hungry , more susceptible to interference from other radiation sources , and can affect the scene . Cameras on the other hand , are much cheaper , passive , and capture much more information at a higher frame-rate . However , it is both hard and computationally expensive to get accurate depth and point clouds from cameras . The classic approach for stereo layout estimation contains two steps . The first step is to generate a BEV feature map by an orthographic projection of the point cloud generated using stereo images . The second step is bird ’ s eye view semantic segmentation using the projected point cloud from the first step . This approach is limited by the estimated point cloud accuracy because the error in it will propagate to the layout estimation step . In this paper , we show that explicit depth estimation is actually neither sufficient nor necessary for good layout estimation . Estimating accurate depth is not sufficient because many areas in the 3D space can be occluded partially , e.g . behind a tree trunk . 1The code and the synthesized dataset will be made public upon the acceptance of this paper . However , these areas can be estimated by combining spatial reasoning and geometric knowledge in bird ’ s eye view representation . Explicitly estimating accurate depth is also not necessary because layout estimation can be done without estimating the point cloud . Point cloud coordinate accuracy is limited by the 3D to 2D BEV projection and rasterization . For these reasons , having an effective bird ’ s eye view representation is very important . SBEVNet is built upon recent deep stereo matching paradigm . These deep learning based methods have shown tremendous success in stereo disparity/depth estimation . Most of these models ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) generate a 3-dimensional disparity feature volume by concatenating the left and right images shifted at different disparities , which is used to make a cost volume containing stereo matching costs for each disparity value . Given a location in the image and the disparity , we can get the position of the corresponding 3D point in the world space . Hence , every point in the feature volume and cost volume corresponds to a 3D location in the world space . The innovation in our approach comes from the observation : it is possible to directly use the feature volume for layout estimation , rather than a two step process , which uses the point cloud generated by the network . We propose SBEVNet , an end-to-end neural architecture that takes a pair of stereo images and outputs the bird ’ s eye view scene layout . We first project the disparity feature volume to the BEV view , creating a 2D representation from the 3D volume . We then warp it by mapping different disparities and the image coordinates to the bird ’ s eye view space . In order to overcome the loss of fine grained information imposed by our choice of the stereo BEV feature map , we concatenate a projection of the original images and deep features to this feature map . We generate these projected features by applying inverse perspective mapping ( IPM ) ( Mallot et al. , 1991 ) to the input image and its features , choosing the ground as the target plane We feed this representation to a U-Net in order to estimate the BEV semantic map of the scene . In order to perform inverse perspective mapping , we require information about the ground in the 3D world space . Hence we also consider the scenario where we perform IPM during the training time and not the inference time . Here , during the training time , we use cross modal distillation to transfer knowledge from IPM features to the stereo features . SBEVNet is the first approach to use an end-to-end neural architecture for stereo layout estimation . We show that SBEVNet achieves better performance than existing approaches . SBEVNet outperforms all the baseline algorithms on KITTI ( Geiger et al. , 2013 ) dataset and a synthetically generated dataset extracted from the CARLA simulator ( Dosovitskiy et al. , 2017 ) . In summary , our contributions are the following : 1 . We propose SBEVNet , an end-to-end neural architecture for layout estimation from a stereo pair of images . 2 . We learn a novel representation for BEV layout estimation by fusing projected stereo feature volume and fine grained inverse perspective mapping features . 3 . We evaluate SBEVNet and demonstrate state-of-the-art performance over other methods by a large margin on two datasets – KITTI dataset and our synthetically generated dataset using the CARLA simulator . 2 RELATED WORK . To the best of our knowledge , there is no published research paper for estimating layout given a pair of stereo images . However , there are several works tackling layout estimation using a single image or doing object detection using stereo images . In this section , we review the most closely related approaches . Monocular Layout Estimation MonoLayout ( Mani et al. , 2020 ) uses an encoder-decoder model to estimate the bird ’ s eye view layout using a monocular input image . They also leverage adversarial training to produce sharper estimates . MonoOccupancy ( Lu et al. , 2019 ) uses a variational encoderdecoder network to estimate the layout . Both MonoLayout and MonoOccupancy do not use any camera geometry priors to perform the task . Schulter et al . ( 2018 ) uses depth estimation to project the image semantics to bird ’ s eye view . They also use Open Street Maps data to refine the BEV images via adversarial learning . Wang et al . ( 2019c ) uses ( Schulter et al. , 2018 ) to estimate the parameters of the road such as lanes , sidewalks , etc . Monocular methods learn strong prior , which does not generalize well when there is a significant domain shift . Stereo methods learn weak-prior plus geometric relationship , which can generalize better . Deep Stereo Matching Several methods like ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) extract the features of the stereo images and generate a 3D disparity feature volume for disparity/depth estimation . They use a 3D CNN on the feature volume to get cost volume to perform stereo matching . PSMNet ( Chang & Chen , 2018 ) uses a spatial pyramid pooling module and a stacked hourglass network to further improve the performance . High-res-stereo ( Yang et al. , 2019 ) uses a hierarchical model , creating cost volumes at multiple resolutions , performing the matching incrementally from over a coarse to fine hierarchy . Bird ’ s Eye View Object Detection Several approaches like Yang et al . ( 2018 ) ; Shi et al . ( 2019 ) use LiDAR to perform 3D object detection . Pseudo-lidar ( Wang et al. , 2019a ) and pseudo-LiDAR++ ( You et al. , 2019 ) use stereo input to first generate a 3D point cloud and then use a 3D object detection network ( Ku et al. , 2018 ; Yang et al. , 2018 ; Shi et al. , 2019 ) on top . BirdGAN ( Srivastava et al. , 2019 ) maps the input image to bird ’ s eye view using adversarial learning . The closest work to our approach is DSGN ( Chen et al. , 2020 ) which constructs a depth feature volume and map it to the 3D space which is then projected to bird ’ s eye view to perform object detection . The task of object detection is of sparse prediction , whereas layout estimation is of dense fine granularity prediction . Hence we introduced IPM to fuse low level detail with the stereo information to improve the performance of layout estimation . 3 OUR METHOD . This section describes the detailed architecture of our proposed framework . SBEVNet is built upon recent deep stereo matching paradigms and follows the rules of multi-view camera geometry . An overview of the SBEVNet is summarized in Figure 1 . 3.1 PROBLEM FORMULATION . In this paper , we address the problem of layout estimation from a pair of stereo images . Formally , given a reference camera image IR and a target camera image IT both of size H×W×3 , the camera intrinsics K , and the baseline length Tb , we aim to estimate the bird ’ s eye view layout of the scene . In particular , we estimate the BEV semantic map of size Nx × Ny × NC within the rectangular range of interest area ( xmin , xmax , ymin , ymax ) in front of the camera . Here H is image height , W is image width , and 3 indicates RGB channels . Nx and Ny are the number of horizontal cells and vertical cells respectively in bird ’ s eye view . NC is the number of semantic classes . This BEV semantic map contains the probability distribution among all semantic classes at each cell of the layout . We assume that the input images are rectified . 3.2 FEATURE EXTRACTION . The first step for SBEVNet is to extract features FR and FT of size H ′ ×W ′ × C for the reference image and the target image respectively . This is done by passing IR and IT through a convolutional encoder with shared weights . This produces multi-channel down-sized feature representations which are next used for building disparity feature volumes . 3.3 DISPARITY FEATURE VOLUME GENERATION . Similar to ( Liang et al. , 2018 ; Khamis et al. , 2018 ; Wang et al. , 2019b ; Sun et al. , 2018 ; Guo et al. , 2019 ; Zhang et al. , 2019 ; Chang & Chen , 2018 ; Kendall et al. , 2017 ) we form a disparity feature volume V by concatenating the features FR and F dT , where F d T is FT shifted horizontally by a disparity of d pixel , resulting in a 3D volume of size H ′ × W ′ × D × 2C . We then pass the feature volume through a series of 3D convolution layers with skip connections to learn higher level features . This feature volume at each d ∈ { 0 , 1 , · · · , D − 1 } contains a representation of the 3D world at the depth corresponding to the disparity d. Rather than using this feature volume to do disparity estimation , we project and warp it to form a bird ’ s eye view representation in the next step .

Summary: The paper proposes an end-to-end network for layout estimation from stereo images. The approach is built off previous stereo matching networks, which built and process a 3D disparity volume. The stereo estimate is used to project image features into a birds-eye-view representation which is processed using a U-net which predicts a semantic scene layout. The approach is evaluated on the KITTI and Carla generated datasets.