SBEVNet: End-to-End Deep Stereo Layout Estimation
Authors: Divam Gupta
Wei Pu
Trenton Tabor
Jeff Schneider
Venue:  (2021)
Keywords: Layout Estimation
Deep Stereo
Computer Vision

Abstract: Accurate layout estimation is crucial for planning and navigation, for robotics applications such as self driving. In this paper, we introduce stereo bird's eye view network SBEVNet, a novel supervised end-to-end framework for estimation of bird's eye view layout from a pair of stereo images. Although our network reuses the building blocks from the state-of-the-art deep learning networks for disparity estimation, we show that accurate depth estimation is neither sufficient nor necessary. Instead, the learning of a good internal bird's eye view feature representation is essential for layout estimation. Specifically, we first generate a disparity feature volume using the features of the stereo images and then project it to the bird's eye view coordinates. This gives us coarse grained scene structural information. We also apply inverse perspective mapping (IPM) to map the input images and their features to the bird's eye view. This gives us fine grained texture information. The concatenated IPM features with the projected feature volume creates a rich bird's eye view representation which is capable of spatial reasoning. We use this representation to estimate the BEV semantic map. Additionally, we show that using the IPM features as a supervisory signal for stereo features can give an improvement in performance. We demonstrate our approach on two datasets: KITTI dataset and synthetically generated dataset using the CARLA simulator. For both of the datasets, we establish state-of-the-art performance beyond other baselines.

Under review as a conference paper at ICLR 2021
SBEVNET: END-TO-END DEEP STEREO LAYOUT Es- TIMATION
Anonymous authors Paper under double-blind review
ABSTRACT
Accurate layout estimation is crucial for planning and navigation in robotics ap- plications, such as self-driving. In this paper, we introduce the Stereo Bird's Eye View Network (SBEVNer), novel supervised end-to-end framework for estima- tion of bird's eye view layout from a pair of stereo images. Although our network reuses some of the building blocks from the state-of-the-art deep learning net- works for disparity estimation, we show that explicit depth estimation is neither sufficient nor necessary. Instead, the learning of a good internal bird's eye view feature representation is effective for layout estimation. Specifically, we first gen- erate a disparity feature volume using the features of the stereo images and then project it to the bird's eye view coordinates. This gives us coarse- -aarsse-gneeeddd informa- tion about the scene structure. We also apply inverse perspective mapping (IPM) to map the input images and their features to the bird's eye view. This gives us fine-grained texture information. Concatenating IPM features with the projected feature volume creates rich bird's eye view representation which is useful for spatial reasoning. We use this representation to estimate the BEV semantic map. Additionally, we show that using the IPM features stereo features can give an improvement in performance. We demonstrate our ap- proach on two datasets: the KITTI (Geiger al. 2013) dataset and a synthetically generated dataset from the CARLA (Dosovitskiy et al. (017)) simulator: For both of these datasets, we establish state-of-the-arr performance compared to baseline techniques.
supervisory signal for
INTRODUCTION
Layout estimation is an extremely important task for navigation and planning in numerous robotics applications such as autonomous driving cars. The bird's eye view (BEV) layout is a semantic occupancy map containing per pixel class information, e.g. road, sidewalk, cars, vegetation, etc. The BEV semantic map is important for planning the path of the robot in order to prevent it from hitting objects and going impassable locations.
In order to generate a BEV layout, we need 3D information about the scene. Sensors such as LiDAR (Light Detection And Ranging) can provide accurate point clouds. The biggest limitations of LiDAR are high cost, sparse resolution, and low scan-rates. Also, as an active sensor LiDAR is more power hungry, more susceptible interference from other radiation sources, and affect the scene. Cameras on the other hand, are much cheaper, passive, and capture much more information higher frame-rate. However, both hard and computationally expensive to get accurate depth and point clouds from cameras.
The classic approach for stereo layout estimation contains two steps. The first step is to generate BEV feature map by an orthographic projection of the point cloud generated using stereo images. The second step bird's eye view semantic segmentation using the projected point cloud from the first step. This approach is limited by the estimated point cloud accuracy because the error in it will propagate to the layout estimation step. In this paper, we show that explicit depth estimation is actually neither sufficient nor necessary for good layout estimation. Estimating accurate depth is not sufficient because many areas in the 3D space can be occluded partially, e.g. behind a tree trunk.
The code and the synthesized dataset will he made public upon the acceptance of this paper.

---

Under review as a conference paper at ICLR 2021
However, these areas be estimated by combining spatial reasoning and geometrie knowledge in bird's eye view representation. Explicitly estimating accurate depth is also not necessary because layout estimation can be done without estimating the point cloud. Point cloud coordinate accuracy is limited by the 3D to 21) BEV projection and rasterization. For these reasons, having an effective bird's eye view representation is very important.
SBEVNer is built upon recent deep stereo matching paradigm. These deep learning based methods have shown tremendous success stereo disparity/depth estimation. Most of these models (Liang et al. 2018 Khamis et al. 2018 g Wang et al., 2019b [un et al.) 2018) Guo et al. 2019) Zhang et al., 2019 Chang & Chen 2018 8 Kendall al. (017) generate 3 d-mmennsinnal disparity feature volume by concatenating the left and right images shifted at different disparities, which is used to make a cost volume containing stereo matching costs for each disparity value. Given a location in the image and the disparity, we can get the position of the corresponding 3D point in the world space. Hence, every point in the feature volume and cost volume corresponds The innovation in our approach comes from the observation: it is possible to directly use the feature volume for layout estimation, rather than a two step process, which uses the point cloud generated by the network. We propose SBEVNet, end-to-end neural architecture that takes pair of stereo images and outputs the bird's eye view scene layout. We first project the disparity feature volume to the BEV view, creating a 2D representation from the 3D volume. We then warp it by mapping different disparities and the image coordinates to the bird's eye view space. In order to overcome the loss of fine grained information imposed by our choice of the stereo BEV feature map. we concatenate a projection of the original images and deep features to this feature map. We generate these projected features by applying inverse perspective mapping (IPM) (Mallot et al. [99T] the input image and its features, choosing the ground as the target plane We feed this representation to a U-Net in order to estimate the BEV semantic map of the scene.
3D location in the world space.
In order to perform inverse perspective mapping, we require information about the ground in the 3D world space. Hence we also consider the scenario where we perform IPM during the training time and not the inference time. Here, during the training time, we use cross modal distillation to transfer knowledge from IPM features to the stereo features.
SBEVNet is the first approach to use an end-to-end neural architecture for stereo layout estimation. We show that SBEVNet achieves better performance than existing approaches. SBEVNer outper- forms all the baseline algorithms KITTI (Geiger et al. 2013) dataset and synthetically gen- erated dataset extracted from the CARLA simulator (Dosovitskiy et al. (017). In summary, our contributions are the following:
1. We propose SBEVNet, an end-to-end neural architecture for layout estimation from stereo pair of images.
2. We learn a novel representation for BEV layout estimation by fusing projected stereo fea- ture volume and fine grained inverse perspective mapping features.
3. We evaluate SBEVNet and demonstrate state-of-the-ar performance over other methods by a large margin on two datasets KITTI dataset and our synthetically generated dataset using the CARLA simulator.
2 RELATED WORK
To the best of our knowledge, there no published research paper for estimating layout given a pair of stereo images. However, there are several works tackling layout estimation using single image or doing object detection using stereo images. In this section, we review the most closely related approaches.
Monocular Layout Estimation MonoLayout (Mani et al. 2020) uses an encoder-decoder model to estimate the bird's eye view layout using monocular input image. They also leverage adversarial training to produce sharper estimates. MonoOccupancy (Lu et al.) 2019) uses a variational encoder- decoder network to estimate the layout. Both MonoLayout and MonoOccupancy do not use any camera geometry priors to perform the task. [Schulter et al. (2018) uses depth estimation to project the image semantics to bird's eye view. They also Open Street Maps data to refine the BEV images via adversarial learning. Wang et al. (2019c) uses (Schulter et al.) 2018) to estimate the parameters of the road such lanes, sidewalks, etc. Monocular methods learn strong prior, which
2

---

Under review as a conference paper at ICLR 2021
Inverse Perspective Mapping
IPM RGB
N
BEV Representation
IPM Features
Inverse Perspoctive Mupping
Stereo BEV Features
U-Net
Warping to world space
Feature Extractor
Reduced Feature Valume
H"x Wixwwxx
D
Reference Image
1
Reduction acnoss height
N, XxN,
Feature Exurctor
Disparity Feature Volume
Semantic-mmp
Visibility mask
Masked cross entropy loss
Target Image
Figure SBEVNel overview. We first extract the image features given the target and reference image. Using the pair of features, create disparity feature volume. We then reduce the disparity feature volume along with the height and warp it in the bird's eye view layout space. On a parallel branch of the network, its features. We concatenate the IPM RGB, IPM feature, and the stereo BEV features. The BEV representation then used estimate the semantic map through U-Net. Visibility mask is used to apply the supervised loss only at the locations in the BEV which are in the view of the front camera.
apply inverse perspective mapping (IPM) the reference image and
does not generalize well when there is a significant domain shift. Stereo methods learn weak-prior plus geometric relationship, which can generalize better.
Deep Stereo Matching Several methods like (Liang al. 2018 Khamis et al. 2018 Wang al, 2019b Sun et al.) 2018 Guo et al.) 2019 Zhang al. 2019 2 & Chen 2018: Kendall et al, 2017) extract the features of the stereo images and generate a 3D disparity feature volume for dis- parity/depth estimation. They use a 3D CNN on the feature volume to get cost volume to perform stereo matching. PSMNet (Chang & Chen 2018) uses a spatial pyramid pooling module and a stacked hourglass network to further improve the performance. High-res-stereo (Yang et al.) 2019) uses a hierarchical model, creating cost volumes at multiple resolutions, performing the matching incrementally from over coarse to fine hierarchy.
Bird's Eye View Object Detection Several approaches like Yang et al. (2018)
: Shi et al. (2019) use LiDAR perform 3D object detection. Pseudo-lidar (Wang al. 2019a) and pseudo-LiDAR+
You et al. 2019) use stereo input to first generate a 3D point cloud and then use a 3D object detection network (Ku al. 2018 Yang et al. 2018 Shi et al. 2019) on top. BirdGAN (Srivastaval et al. 2019) maps the input image to bird's ' eye view using ad adversarial learning. The closest work our approach is DSGN (Chen et al. (020) which constructs a depth feature volume and map to the 3D space which is then projected to bird's eye view to perform object detection. The task of object detection of sparse prediction, whereas layout estimation is of dense fine granularity prediction. Hence we introduced IPM to fuse low level detail with the stereo information to improve the performance of layout estimation.
3
OUR METHOD
This section describes the detailed architecture of proposed framework. SBEVNet is built upon recent deep stereo matching paradigms and follows the rules multi-view camera geometry. An overview of the SBEVNet is summarized in Figure

---

Under review as a conference paper at ICLR 2021
3.1 PROBLEM FORMULATION
In this paper, we address the problem of layout estimation from a pair of stereo images. Formally, given a reference camera image R and target camera image Ir both of size H W x3, the camera intrinsics K, and the baseline length To, we aim to estimate the bird's 's eye view layout of the scene. In particular, we estimate the BEV semantic map of size N₂ x N2 Ne within the rectangular range of interest area (Zmin Zmars Imin: Mmarx) ) in front of the camera. Here H is image height, W is image width, and indicates RGB channels. N₂ and Ng are the number of horizontal cells and vertical cells respectively in bird's eye view. Nc is the number of semantic classes. This BEV semantic map contains the probability distribution among all semantic classes at each cell of the layout. We assume that the input images are rectified.
3.2 FEATURE EXTRACTION
The first step for SBEVNet is to extract features FR and Fr of size H' x W x C for the reference image and the target image respectively. This done by passing In and Ir through convolutional encoder with shared weights. This produces multi-channel down-sized feature representations which are next used for building disparity feature volumes.
3.3 DISPARITY FEATURE VOLUME GENERATION
Sun et al. 2018 Guo al. 2019 Zhang et al. 2019: Chang & Chen 2018 Kendall et al. 2017) we form a disparity feature volume V by concatenating the features Fn and Fÿ. where 1 is Fr shifted horizontally by disparity of d pixel, resulting in 3D volume of size H' x W x D x 2C. We then pass the feature volume through a series of 3D convolution layers with skip connections to learn higher level features. This feature volume at each d E 1, -. D - 1} contains a representation of the 3D world at the depth corresponding to the disparity Rather than using this feature volume to do disparity estimation, we project and warp it to form bird's eye view representation in the next step.
Similar to (Liang et al. 2018 Khamis et al. 2018, Wang et al. 2019b;
3.4 BIRD'S ' EYE VIEW REPRESENTATION
The bird's eye view representation is composed of two parts 1) The stereo BEV representation which is derived from the disparity feature volume, 2) The IPM BEV representation which is the re- sult of applying inverse perspective mapping on the reference image and the features of the reference image. These two parts are concatenated to form the final bird's eye view representation.
3.4.1 STEREO BEV REPRESENTATION
The disparity feature volume generated is widely used to estimate depth/disparity in the stereo image pairs. But this feature volume contains a lot of information about the 3D) scene which can be used for other tasks as well. Each point in the disparity feature volume corresponds to a point in the 3D world space. We first need to map the 3D feature volume to a 2D feature map containing information of the bird's 's eye view. we do max/average pooling along height dimension, a certain degree of the height information is lost quickly before being extracted for our task, which is not desirable. Considering height information a good prior for layout estimation but we don't need to recover it explicitly, we concatenate the feature volume along the height, creating a 2D image of size W' x D x 2CH'. We then use 2D convolutions to generate the reduced feature volume of size W x D" x C". This reduced feature volume does not spatially match with the bird's eye view layout. Hence. the reduced feature volume, transforming it to a feature map of size N₂ x N, x C' in the bird's eye view space. Given the disparity d, position in the image along width u, camera parameters f, Ca. cy. and stereo baseline length Tz, we can find the coordinates in the bird's eye space x' and 1/ as follows:
warp
Cz) T
d
f T
(2)
d
4

---

Under review as a conference paper at ICLR 2021
The 2D origin of the bird's eye view is co-located with the reference camera. An example visualiza- tion of the layout in the disparity volume space shown in Figure After mapping the coordinates to the BEV space, map them to grid of size N₂ x N2 giving the stereo BEV representation Rateren-
3.4.2 IPM BEV REPRESENTATION
The stereo BEV representation contains structural information for the bird's eye view space. Due to the refinement and reduction of the feature volume, the fine grained details are excluded by design. To circumvent that, we need to fuse the low level features to the stereo BEV features, while maintaining geometric consistency.
In order to fuse the image features to the stereo BEV features at the correct locations, we need to warp the image features the BEV space. We apply inverse perspective mapping on the reference image and the features of the reference image to do that.
A point in the image In correspond to multiple points in the 3D world space due to perspective projection, but there is a single point which also intersects with the ground plane. Let 2 = by-tb be the equation of the ground plane the world space. Given the input image coordinates (u, v) and camera parameters fCCa, Cy we can find the coordinates in the bird's eye space x' and y as follows:
CU CC2
x'
(3)
aC2 au - bf Cy +
cf
(4)
ac2 - au - bf Cy +
This can be easily derived by combining the camera projection equation with the equation of the ground plane. For many applications, the ground is either planar or can be approximated by a plane. This is also equivalent to computing homography // between the ground plane and the image plane of the layout and then applying the transformation. We can have the parameters of the plane a, b, and pre-determined if the placement of the camera with respect to the ground is known, which is the case for many robotics applications. We can also determine a, and e by using stereo depth and semantic segmentation network for the road/ground class.
Examples IPM on the input images is shown in Figure 3 in the appendix. We apply the inverse perspective transform both the input image and the features of the input image transform them to the bird's eye view space:
RiPM fear IPM(FR)
(5)
RaPMiimg (P)()R)
(6)
They are then concatenated with the stereo BEV representation form the combined BEV repre- sentation:
RAEV [RPM feat; RIPM img' Risereo]
3.4.3 IPM FOR CROSS MODAL DISTILLATION
There can be use-cases where we cannot do inverse perspective mapping during inference time, due the unavailability of the ground information. Hence, we consider the case where IPM is only available during the training time. We can think of the IPM features and the stereo features as different modalities and apply cross modal distillation (Gupta et (2016)) across them, and transfer knowledge from IPM features to the stereo features. Hence, we use the IPM BEV representation as a supervisory signal for the stereo BEV features. This forces the stereo branch of the model to implicitly learn the fine grained information learned by the IPM features. Rather than concatenating the IPM BEV features with the stereo features, we minimize the distance between them. We call this variant of SBEVNet SBEVNet-CMD (SBEVNer cross modal distillation.. During the training time, the IPM BEV features and the stereo features are used to generate the BEV semantic maps.
IPM
U-Net( RiPM.te))
5

---

Under review as a conference paper at ICLR 2021
C U-Net U-Net(RReereo) Rteeeo))
(9)
This ensures both IPM BEV features and stereo BEV features learn meaningful information. We jointly minimise the 1 distance between first K channels of the features.
KT
fep K] Raterol |: K]]l,,
(10)
By this, we ensure that the stereo model can learn information that not in the IPM features. In our experiments, found this to yield better results compared to the approach of minimizing the Lı distance between all the channels of the features. During test time, we only use the stereo features to get the BEV layout. Our experiments show that the stereo model with cross modal distillation performs better than the stereo model without cross modal distillation.
3.4.4 LAYOUT GENERATION
We can generate the semantic map by inputting the BEV features to semantic segmentation net- work. We pass the concatenated stereo BEV feature map and IPM BEV feature map U-Net Ronneberger et al. 2015) network to generate the semantic map C.
C = U-Net(RBBE)
(11)
Some areas in the layout may not be in the view of the front camera, e.g. things behind wall. That is why it is not good idea to penalize the model for the wrong prediction for those areas. Hence, use visibility mask mask the pixel-wise loss, applying it only the pixels which are in the field of view. This mask is generated during the ground truth generation process by using ray- ray-tracing on the point cloud determine which are in the field of view. For visibility mask V. V2 the pixel is in the view of the input image, and otherwise. For the loss, we use pixel-wise categorical cross entropy loss follows:
Lr
Σ V1 MT-CCE(C,, CiPt) iEP
(12)
where C is ground truth. The total loss for SBEVNeI-CMD is the sum of supervision loss from the two feature maps and the L1 distance minimization.
L
Σ V, CCE(C! İEP
Ct)) +
Σ
V P7-CCE(C, (.tlero C.ti) + LKT
(13)
NE
EXPERIMENTS
4.1 DATASETS
CARLA dataset: We use the CARLA (Dosovitskiy et al. 2017) simulator to generate a synthetic dataset, containing 4,000 and 925 training and testing data points respectively. The bounds of the layout with respect to the camera are -19 to 19 meters in æ direction and to 39 meters in the direction.
KITTI dataset: We also evaluate SBEVNet on the odometery subset of the KITTI (Geiger et al. dataset. We use the SemanticKITTI Behhl al. (2019) dataset for labeled ground truth.
2013
More details about the datasets can be found in the Appendix.
4.2
EVALUATION METRICS
As not all the regions of the ground truth layout are visible from the camera, we only consider pixels of the layout which are in the field of view. For evaluating the semantic map, we use macro averaged intersection over union (IoU) scores for the layout pixels which are in the visibility mask. We report the IoU scores for each semantic class separately.
4.3 COMPARED METHODS
There are no previously reported quantitative results for the task stereo layout estimation in our setting. Thus, evaluate appropriate baselines which prior works extended our task.

---

Under review as a conference paper at ICLR 2021
Method
mloU
Road
Vegetation
Cars
Sidewalk
Building
Pseudo-LiDAR segmentation
25.63
37.64
16.40
35.15
25.44
13.50
Pseudo-LiDAR BEV U-Net
36.61
63.55
31.87
45.64
29.97
12.01
IPM + BEV U-Net
32.30
66.36
15.24
41.37
32.77
5.78
MonolLayout
22.16
52.88
9.00
16.36
23.17
9.41
MonoLayout + depth
21.85
52.94
9.95
14.07
23.02
9.31
MonoOecupancy depth
29.49
67.96
16.56
7.66
36.35
18.91
SBEVNer only stereo
36.10
64.74
31.85
39.76
30.01
14.14
SBEVNe1 stereo + RGB IPM
39.77
65.01
33.20
47.88
33.24
19.53
SBEVNel stereo features IPM
42.29
71.29
29.79
51.97
38.46
19.95
SBEVNet-CMD SBEVNet
40.10
69.07
32.71
45.45
35.16
18.08
44.36
72.82
32.07
55.32
40.69
20.78
SBEV VNet Ensemble 47.92
75.36
35.33
60.17
44.25
24.47
Table 1: Quantitative results of semantic layout estimation on the CARLA dataset.
Input
Ground truth
PseudoLidar U-Net
SBEVNet only stereo
SBEVNet
Vegetation
Road
Sidewalk
Cars
Building
Figure 2:
Qualitative results the test set of the CARLA and the KITTI dataset. The major mistakes in the predictions are annotated by blue rectangle.
We first compare method with Pseudo-lidar (Wang et al. 2019a). where we use PSMNet to generate a 3D point cloud. In Pseudo-LiDAR segmentation we project the semantic segmentation the bird's eye view. In Pseudo-LiDAR BEV U-Net, the RGB
of the front view adn project 3D point projected in the BEV space is fed as an input to the U-Net segmentation network. IPM BEV U-Net inverse perspective mapping is applied to the input image, which fed input the

---

Under review as a conference paper at ICLR 2021
Method
mloU
Road
Sidewalk
Cars
Building
Vegetation
Pseudo-LiDAR + segmentation
18.69
35.51
14.56
13.50
12.64
17.26
Pseudo-LiDAR BEV U-Net
28.97
61.83
21.36
5.55
12.74
43.38
IPM+ BEV U-Net
34.93
68.40
26.65
28.49
4.53
46.57
MonoLL
25.19
64.36
20.53
2.43
2.59
36.05
MonoLayoutt + depth
21.48
55,80
16.19
1.91
3.03
30.46
MonoOccupancy + depth
29.16
70.52
22.17
7.11
5.25
40.77
SBEVNer only stereo
50.01
78.41
40.16
41.96
30.45
59.05
SBEVNeI stereo + RGB IPM
49.56
78.37
39.83
42.47
28.34
58.80
SBEVNet stereo features IPM
50.60
80.16
41.08
43.64
29.19
58.92
SBEVNeI-CMD
50.73
80.59
41.67
43.16
29.13
59.37
SBEVNet
51.36
80.23
41.86
42.81
31.35
59.43
SBE VNet Ensemble
53.85
82.22
45.70
44.97
34.54
61.83
Table 2: Quantitative results of semantic layout estimation on the KITTI dataset.
U-Net segmentation network. In MonoLayout (Mani al. 2020) + depth and MonoOccupancy [Lu et al. 2019) depth, we concatenate the input RGB image with the depth to train MonoLayout and MonoOccupancy model. Detailed description of the baselines can be found in the appendix.
We also evaluate some variations of our model to perform ablation studies. In SBEVNet only stereo we exclude the IPM features and only use features derived from the feature volume. To gauge the importance of IPM on RGB images and features, (SBEVNet stereo + RGB IPM) and IPM only on the features the input image (SBEVNet stereo features IPM). We also evaluate the cross modal distillation model SBEVNet-CMMD Finally, evaluate our complete model (SBEVNet ) where we use stereo features and IPM on both RGB image and its features. We also evaluate SBEVNet Ensemble where take ensemble of SBEVNet with the same architecture but different initialization seeds.
also try applying IPM only on RGB images
4.4 EXPERIMENTAL RESULTS
We report the IoU scores of all the methods on the CARLA and KITTI (Geiger et al. 2013) dataset from the tables, SBEVNer achieves superior
in Table D and Table 2 respectively. As performance on both the datasets. We also observe the increase in performance if we use both stereo information and inverse perspective mapping. IPM yields greater increase in performance in the CARLA (Dosovitskiy et al., 2017) dataset because the ground is perfectly flat. If we use only RGB IPM along with stereo, the results are slightly worse on the KITTI dataset because the ground is not perfectly planar. We see that degradation does not persist if we also use IPM on the image features. For the KITTI dataset, we see sharp improvement over pseudo-LiDAR approaches because of inaccurate depth estimation. On the other hand, model does not depend explicit depth data/model. The results of MonoLayout (Mani et al. 2020) and MonoOccupancy (Lu et al., 2019) are inferior due to lack of any camera geometry priors in the network. We also show the qualitative results on the test set CARLA (Dosovitskiy et al. 2017) and KITTI (Geiger et al, 2013) dataset in Figure We see that in certain regions SBE VNet gives outputs closer the ground truth. More details and analysis of the results can be found in the appendix.
5 CONCLUSION
In this paper we proposed SBEVNet, an end-to-end network to estimate the bird's eye view layout using pair of stereo images. We observe improvement in the loU scores compared with approaches that are not end-to and-to-end or do not use geometry. We also showed that combining inverse perspective mapping with the projected disparity feature volume gives better performance. We also show that, using cross modal distillation to transfer knowledge from IPM features to the stereo features gives improvement in results.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. SemanticKITTI: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the IEEE International Conference on Computer Vision, pp. 9297-9307, 2019,
Jia-Ren Chang and Yong-Sheng Chen. Pyramid stereo matching network. Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5410-5418, 2018.
Yilun Chen, Shu Liu, Xiaoyong Shen, and Jiaya Jia, DSGN: Deep stereo geometry network for 3d object detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12536-12555 2020.
Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun. CARLA: An open urban driving simulator. arXiv preprint arXiv:/711.03938 2017.
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets roboties: The KITTI dataset. The International Journal of Robotics Research, 32(1 2013.
Xiaoyang Guo, Kai Yang, Wukui Yang, Xiaogang Wang, and Hongsheng Li. Group-wise corre- lation stereo network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3273-3282, 2019.
Saurabh Gupta, Judy Hoffman, and Jitendra Malik. Cross modal distillation for supervision transfer. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2827- 2836, 2016.
Alex Kendall, Hayk Martirosyan, Saumitro Dasgupta, Peter Henry, Ryan Kennedy, Abraham Bachrach, and Adam Bry. End-to-end learning of geometry and context for deep stereo regression. In Proceedings of the IEEE International Conference on Computer Vision, pp. 66-75, 2017.
Sameh Khamis, Sean Fanello, Christoph Rhemann, Adarsh Kowdle, Julien Valentin, and Shahram Izadi. StereoNet: Guided hicrarchical refinement for real-time edge-aware depth prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 573-590, 2018.
Jason Ku, Melissa Mozifian, Jungwook Lee, Ali Harakeh, and Steven L Waslander. Joint 3D pro- posal generation and object detection from view aggregation. In 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1-8. IEEE, 2018.
Zhengfa Liang, Yiliu Feng, Yulan Guo, Hengzhu Liu, Wei Chen, Linbo Qiao, Li Zhou, and Jianfeng Zhang. Learning for disparity estimation through feature constancy. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 281 2811-2820, 2018.
Chenyang Lu, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. Monocular se- mantic occupancy grid mapping with convolutional variational encoder-decoder networks. IEEE Robotics and Automation Letters, 4(2):44 452, 2019.
Hanspeter Mallot, Heinrich Bülthoff, JJ Little, and Stefan Bohrer. Inverse perspective mapping simplifies optical flow computation and obstacle detection. Biological cybernetics, 64(3):177- 185. 1991.
Kaustubh Mani, Swapnil Daga, Shubhika Garg, Sai Shankar Narasimhan, Madhava Krishna, and Krishna Murthy Jatavallabhula. MonoLayout: Amodal scene layout from a single image. The IEEE Winter Conference on Applications of Computer Vision, pp. 1689-1697, 2020.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Samuel Schulter, Menghua Zhai, Nathan Jacobs, and Manmohan Chandraker. Learning to look around objects for view representations of outdoor scenes. In Proceedings of the European Conference Computer Vision (ECCV), pp. 787-802, 2018.

---

Under review as a conference paper at ICLR 2021
Shaoshuai Shi, Xiaogang Wang, and Hongsheng Li. PointRCNN: 3D object proposal generation and detection from point cloud. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770-779, 2019.
Siddharth Srivastava, Frederic Jurie, and Gaurav Sharma. Learning 2D to 3D lifting for object detection in 3d for autonomous vehicles. arXiv preprint arXiv: arXiv: 190.008494 2019.
Deqing Sun, Xiaodong Yang. Ming- Yu Liu, and Jan Kautz. PWC-Net: CNNs for optical flow using pyramid, warping, and cost volume. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2018.
Yan Wang, Wei-Lun Chao, Divyansh Garg, Bharath Hariharan, Mark Campbell, and Kilian Q Wein- berger. Pseudo-LiDAR from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1445-8453 2019a.
Yan Wang, Zihang Lai, Gao Huang, Brian H Wang, Laurens Van Der Maaten, Mark Campbell, and Kilian Q Weinberger. Anytime stereo image depth estimation on mobile devices. In 2019 International Conference Robotics and Automation (ICRA), pp. 5893-5900. IEEE, 2019b.
Ziyan Wang, Buyu Liu, Samuel Schulter, and Manmohan Chandraker. A parametric top-view rep- resentation of complex road scenes. In Proceedings of the IEEE Conference Computer Vision and Puttern Recognition, pp. 10325-10333, 2019c.
Bin Yang, Wenjie Luo, and Raquel Urtasun. PIXOR: Real-time 3d object detection from point clouds. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 7652-7660, 2018.
Gengshan Yang, Joshua Manela, Michael Happold, and Deva Ramanan. Hierarchical deep stereo matching on high-resolution images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5515-5524, 2019.
Yurong You, Yan Wang, Wei-Lun Chao, Divyansh Garg, Geoff Pleiss, Bharath Hariharan, Mark Campbell, and Kilian Q Weinberger. LiDuRo-LiAR++: Accurate depth for 3D object detection in autonomous driving. arXiv preprint arXiv: 006606630, 2019.
Feihu Zhang. Victor Prisacariu, Ruigang Yang, and Philip HS Torr. GA-Net: Guided aggregation net for end-to-end stereo matching. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 185-194, 2019.
Zhi-Hua Zhou. Ensemble methods: foundations and algorithms. CRC press, 2012.
A APPENDIX
A.I DATASETS
CARLA dataset: We use the CARLA (Dosovitskiy al. (0117) simulator generate a synthetic dataset. A pair of stereo cameras are placed on a car moving along a set of waypoints. We also randomly change the weather/lighting conditions. We use the point cloud of the simulator's city model to generate the ground truth semantic map. To ensure that we evaluate the generalizability of the models, the training and testing are done on entirely different city models in CARLA. Town01, Town02, Town03, and Town04 are used for training, and Town05 is used for testing. The training set contains 4,000 pairs of stereo images and the testing set contains 926 pairs of stereo images. The classes use for the semantic map road, vegetation, car, sidewalk, and building. The bounds of the layout with respect to the camera are -19 to 19 meters in x direction and I to 39 meters in the y direction.
KITTI dataset: We also evaluate SBEVNet on the publically available KITTI Geiger et al. 2013) dataset. Similar to Mani al. (2020), use the KITTI odometery subset and use the Se- manticKITTI [Behley et al. (2019) dataset for labeled ground truth point clouds. We use the same training/testing split used by Mani et al. (2020), where separate sequences used for training
10

---

Under review as a conference paper at ICLR 2021
Disparity space
Reference image
Bird's eye view space
N
x
Y
transformation
Fym
Figure 3: Illustration of mapping the disparity space to bird's eye view space and inverse perspective mapping. (a) The operation maps different disparities and the BEV space in order to match the ground truth. We also show an example layout warped to the disparity space. (b) The inverse perspective mapping operation maps pixels of the reference image to the BEV space in order to match the ground truth. The same mapping can be applied to the image features as well.
Auxiliary Semumtic-mup
Perspoctive
Mapping
Auxiliary loss
L, minimize loss
Inference time
Features
XNN
Warping world space
Feature Exuractor
Primary Semantic-map
Reduced Feature Volume
H"x wTx
DxC
Reference Image
Reduction across beight
Primary loss
Exuractor
Disparity
Feature
Volume
Target Image W
Figure 4: SBEVNer-CMD overview. We first extract the image features given the target and refer- ence image. Using the pair of features, create stereo BEV representation. During training time, we apply inverse perspective mapping (IPM) on the image features which is used to predict the BEV layout separately. We minimize the L1 distance between first channels of both the BEV feature maps. During inference time, we only use the stereo BEV representation to estimate the semantic map.
and testing. The training set contains 3,278 stereo image pairs and the testing set contains 1,371 stereo image pairs. The classes for the semantic map road, vegetation, car, sidewalk, and

---

Under review as a conference paper at ICLR 2021
building. The bounds of the layout with respect to the camera are -19 19 meters in ø direction and 5 to 43 meters in the y direction.
A.2 COMPARED METHODS
1. Pseudo-LiDAR (Wang et al., 2019a) segmentation: Uses Pscudo-lidar with PSMNet to generate 3D point cloud from the input stereo images which is used to project the semantic segmentation of the front view to the bird's eye view. The PSMNet is trained separately on the respective datasets for better performance.
2. Pseudo-LiDAR (Wang et al., 2019a) + BEV U-Net: The RGB 3D point projected in the BEV aligned with the ground truth layout used train a U-Net segmentation network.
3. IPM + BEV U-Net: Inverse perspective mapping applied to the input image to project to the BEV space which is used to train U-Net segmentation network.
4. MonoLayout (Mani et al. 2020 This baseline uses MonoLayout to generate BEV se- mantic map from single image. Rather than using OpenStreetMap data for adversarial training, we used random samples from the training set itself.
5. MonoLayout (Man et al. 2020) + depth:
The input RGB image concatenated with the depth is used as an input the MonoLayout Model.
6. MonoOccupancy (Lu al. 2019 depth:
The imput RGB image concatenated with the depth is used as an input to the MonoOccupancy Model.
A.3 IMPLEMENTATION DETAILS
We implemented SBEVNet using Pytorch. We use Adam optimizer with the initial learning rate of 0.001 and betas (0.9, 0.999) for training. We use a batch-size of 3 on a Titan X Pascal GPU. We use the same base network which is used in the basic model of PSMNet. The input image size for the CARLA dataset is 512x 288 and the input image size for the KITTI dataset 640x256. We report the average scores according to 8 runs to account for the stochasticity due to random initialization and other non-deterministic operations in the network.
A.4 EXPERIMENTAL RESULTS
A.4.1 ABLATION STUDY
IPM on RGB image For the CARLA dataset, we observe an increase of 3.67 in the mloU score, on concatenating IPM RGB with the stereo features. We observe increase in loU scores for all the classes, with the biggest increase of 8.12 in the cars class. For the KITTI dataset, there is a small decrease of 0.45 in the mIoU score. This is because, the ground is not perfectly planar, hence the IPM RGB images do not exactly align with the ground truth layout.
IPM on image features If apply IPM on the features of the input image and concatenate it with the features from the stereo branch, we see an improvement in both the datasets. The improve- ments in mloU scores are 6.19 and 0.59 for the CARLA and KITTI dataset. The improvement higher compared to the RGB IPM because image features contain higher level information which is transformed to the BEV space.
IPM on both RGB image and image features We see the greatest improvement if we apply IPM on both the RBG image and the features of the RGB image. The improvements in mloU scores are 8.26 and 1.35 for the CARLA and KITTI dataset respectively. This is because the model is able to exploit the different information present in RIPMLteat and Rsserco
Cross modal distillation The performance of SBEVNet- CDM is in between of stereo only SBEVNet and full SBEVNet. We see an improvement of 4.00 and 0.72 in the mloU scores on the CARLA and KITTI dataset, if we train the stereo model using cross modal distillation via IPM features. During inference, the architecture of SBEVNeI-CMM is the same as the stereo only SBEVNet. This shows that CMD is able transfer most of the IPM knowledge to the stereo branch.
Minimizing distance between first K features We also evaluate the approach, where we try mini- mizing the L.1 distance between all the channels of the IPM features and stereo features. We observe
12

---

Under review as a conference paper at ICLR 2021
mloU scores of 32.27 and 50.03 for the CARLA and KITTI dataset respectively. This is worse than the mloU scores achieved by minimizing the distance between first channels. This is because, enforce all stereo branch channels to be the same as IPM branch channel, the stereo branch is unable to learn information that is not present in the IPM features.
A.5 ADDITIONAL ANALYSIS
A.5.1 3D FEATURE VOLUME ANALYSIS
One claim of our approach is that our model learns 3D information without any explicit depth/disparity supervision. To validate this claim, we the learned 3D feature volume to perform disparity estimation. We freeze all the weights and add a small 3D convolution layer to perform dis- parity regression on the learned feature volume. We also observe that the feature volume which is trained with cross modal distillation via IPM performs better at the task of disparity estimation. For the CARLA dataset, we find that the SBEVNet only stereo model has 3-pixel error of 7.92 and with cross model distillation the 3-pixel error goes down to 6.84.
A.5.2 DISTANCE FROM CAMERA
We wish to quantify how our system performs as we move away from the camera. Hence, we plot the IoU scores for the pixels in the BEV layout which are more than a given distance from the camera (Figure 6) and for the pixels which are less than given distance from the camera (Figure For both the KITTI and the CARLA dataset, observe that there is drop in performance as the distance from the camera increases. We also observe that SBEVNer outperforms the stereo only SBEVNet all distances from the camera.
CARLA de laset
KITTI dataset
Mean wwweeeelleeeee artn Imiodn
0.525
SBEVNct only Stores SBEVNet
(mooh)
SREViet
only stores
0.500
0.50
0.475
0.450
0.58
0.425
0.54
0.375
0.50
10
25
30
1.0
20
35
Maximam distarce
Maximam dislance [m]
Figure 5: Performance as a function of maximum distance from the camera. We consider the pixels in the BEV layout which are atmost certain distance away from the camera.
CARLA de Laset
KITTI dataset
0.45
Wean www.rttteeeeeooooooo www.eeeoooaa..........o.. (mvod)
SBEWNet only stores
SBEVNe
0.50
SIEViel
only stores
0.40
0.48
0.46
0.30
0.44
Heam
0.25
1.0
15
20 25
30
35
Ninimum distarce (m]
Minimum distlance (m])
Figure 6: Performance as function of minimum distance from the camera. We consider the pixels in the BEV layout which are atleast certain distance away from the camera.
13

---

Under review as a conference paper at ICLR 2021
A.5.3 AMOUNT OF TRAINING DATA
We also quantify how the performance of SBEVNer changes with the number of data-points in the training set, while keeping the test set same. On the CARLA dataset, with just 10% of the training data, we get mloU score of 26.10 compared the mloU score of 44.36 with all the training data. For both the datasets, performance starts to saturate when we use 100% of the training data. This shows that 3,000-40000 training data points are sufficient for getting the optimal performance from SBEVNet.
CARLA dataset
KITTI cataset
(weets
Meanh wn www. (nreads
0.50
0.45
0.375
0.350
0.40
0.325
0.35
0.300
0.255
0.30
80
100
20
40
60
80
1.0
training data used
training data
Figure 7: Performance of the system function of amount of training data used.
A.5.4 PERFORMANCE EVOLUTION DURING TRAINING
We also perform qualitative analysis (Figure 8 of how the performance of the model changes during training. We observe, during the initial stages of training. the model learns identify very course grained attributes such as the direction of the road. However, there is ambiguity in detailed attributes such as size and exact position. During the later stages of training, the model learns to identify the smaller objects and exact positioning in the BEV space.
Input
Ground truth
Epoch
Epoch
Epoch
Figure Evolution of predicted BEV layouts for different epochs during training
A.5.5 ENSEMBLE
We observe some variance in the performance of the models on training with different random seeds. For SBEVNet observe standard deviation of 2.16 and 2.46 in the mloU scores for the KITTI and CARLA dataset respectively. Due to the diversity in outputs of the individual models (Zhou
2012 ) we improvement in the performance, if take ensemble of individual models.
14

---

Under review as a conference paper at ICLR 2021
We observe absolute improvement 2.49 and 3.56 in the mloU scores for the KITTI and CARLA dataset respectively.
A.5.6 INFERENCE TIME
On NVIDIA Titan X GPU, with the batch size equal to the inference time of stereo only SBEVNer for one input pair is 0. 0.1307s the average. For the full SBEVNer the inference time is 0419s on the average, which is slightly higher than the stereo only model. This inference speed sufficient for majority of robotics applications. The majority of computation is done in processing the 3D feature volume with 3D convolutions.
A.6 RESULTS ON KITTI OBJECT DATASET
We also compare our method with the published numbers on the KITTI Object dataset. We the dataset and annotations provided by Mani et al. (2020). Here, there is only single cars class. We compare our approach with published monocular approaches and 3D object detection approach on pseudo lidar with stereo input. AVOD + pseudo lidar is an object detection method which also uses large sceneflow dataset for pre-training. More details of these baseline models can be found in Mani et al. (2020). Table shows the numbers on the KITTI object split which are provided by Mani ct al. (2020). We observe that our method achieves an improvement in the mloU scores over all the other methods. For segmentation, pixel level mAP is not good metric as is does not consider false negatives. We still report the pixel level mAP scores for reference.
Method
mloU
mAP (pixel level)
ENet Pseudo lidar input (Mnpouttoo)))))h))) PointRCNN
0.24
0.37
PointRCNN Pseudo lidar (Monoddtoh)))))))
0.26
0.43
MonoLayout
0.26
0.41
AVOD Pseudo lidar input(PSMNee (Stereo)
0.43
0.59
Our (Stereo)
0.46
0.59
Table 3: Quantitative results BEV car segmentation on the KITTI object dataset. All the results
except SBEVNet excerpted from Mani al. (2020)
15


Reviews:
[{"review_id": "jyh-d33SwOA", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n## Contributions\n\nThis paper presents SBEVNet, a neural network architecture to estimate the bird's-eye view (BEV) layout of an urban driving scene. Given an image captured by a stereo camera, SBEVNet performs an inverse perspective mapping (IPM) to obtain an initial feature volume, which is further processed to generate the BEV layout. The system is trained end-to-end in a supervised learning setup.\n\n## Strengths\n\n**S1** The problem considered here is very relevant to perception groups in the autonomous driving community. This area has only recently seen work crop up. Approaches like MonoLayout [A], MonoOccupancy [B], and PseudoLidar [C] are closely related to this submission.\n\n**S2** The paper is easy to follow, and provides a majority of the details needed to understand and assess the approach.\n\n**S3** The authors also seem to provide code (and promise a public release), which might help ensure reproducibility.\n\n## Weaknesses\n\nI see a few major and a number of other minor concerns that impact my perception of this paper. I'm hoping the discussion period helps address some of these, and I'm open to revising my score in light of evidence contrary to the following claims.\n\nIt appears that this paper uses MonoLayout [A], MonoOccupancy [B], and PseudoLidar [C] as primary baselines. Much of my review stems from my understanding of [A, B, C] (and my 'surprise' at a few contradictory trends observed in this paper.)\n\n**Problem setup** It is unclear from reading the paper and supplementary material if the problem setup is infact \"amodal\" layout estimation (i.e., if scene points outside of the camera view are predicted in the BEV layout). Approaches like (Schulter et al., 2016) and (Mani et al., 2020) operate in this \"amodal\" setup, while others such as PseudoLidar [C] and (Lu et al., 2019) only predict points that are visible in the input image. Does this approach, for instance, hallucianate hidden intersections and roads? (It seems not, since a visibility mask is explicitly employed in the loss function -- cf. Fig. 1 and Eq. 12, 13).\n\n**MonoLayout baseline** The primary baseline considered in this paper is \"MonoLayout\" (Mani et al., 2020). Upon examining the MonoLayout [A] paper, I find a surprising and troubling trend. This paper reports very poor performances of MonoLayout on the KITTI dataset (the original MonoLayout paper reports mIoU for the \"car\" class to be around 26.08, while the current submission reports 2.43 -- cf. Table 2). I've noted that MonoLayout makes its code and models publicly available (its publicly available pretrained models claim an mIoU of 30.18 for the \"car\" class), as highlighted on their GitHub page. Also, other baselines like \"MonoOccupancy\" have surprisingly low scores in this paper (an order of magnitude), compared to scores reported in the MonoLayout paper. I wonder if there is something different in the experiment and/or training protocols employed in the current work, as opposed to those in the MonoOccupancy and MonoLayout papers? For example, the MonoOccupancy baseline as reported in the MonoLayout paper achieves an mIoU of about 24.16 (for the car class) (MonoLayout paper - Table 1), while the same baseline has a dismal performance (mIoU of 7.11 for car class) in Table 2 of the current manuscript.\n\nThe fact that this performance gap is not explained in the paper makes it hard to analyze the merits of the proposed approach. Save for a single sentence \"The results of MonoLayout ... and MonoOccupancy ... are inferior due to lack of any camera geometry priors in the network\", I've not found any other discussion of this performance gap/discrepancy.\n\nI also find it a tad weird (and unexplained) that the performance of various baselines do not seem to follow a set pattern/trend across the CARLA and KITTI datasets. In the MonoLayout paper, I notice that changing the dataset from KITTI to Argoverse does change absolute mIoU scores a bit, but preserves the ranking of various baselines (i.e., MonoLayout > OFT > MonoOccupancy on both KITTI and Argoverse). In the current submission, the trends seem to be changing across the two datasets (cf. Tables 1, 2).\n\nYet another set of baselines that seem to underperform here are the PseudoLidar variants. In the MonoLayout paper (cf. supplementary material, Table 5), Pseudolidar is evaluated on the KITTI dataset, and the reported mIoU for vehicles is 59, whereas in this paper the best performance on this class achieved by a pseudolidar model is 45.64. Further, the MonoLayout paper's version of the (stereo) Pseudolidar baseline seems to perform quite competetively (mIoU 59.0) to SBEVNet Ensemble (mIoU 60.17 for \"car\", cf. Table 2). This seems to indicate that well-tuned baselines could perhaps achieve better performance?\n\nIn Appendix A.2, the authors seem to indicate that they used a very different process to train MonoLayout (i.e., using random images from the train set as opposed to using OpenStreetMap and/or adversarial training). I suspect this might have resulted in a performance gap?\n\nI feel that OFT [D] could be cited and used as a baseline, particularly to measure layout estimation accuracy for the \"car\" class.\n\n**Qualitative results** Unfortunately, there seems to be a dearth of qualitative result figures to get a better sense of the approach. In particular MonoLayout and MonoOccupancy seem to obtain crisp reconstructions of cars (cf. MonoLayout paper), while in Figure 2., cars are splayed throughout the image in the SBEVNet results. This is also surprising; in my opinion, these results do not adequately substantiate the impressive reported mIoU.\n\n**Missing mAP metric** Other papers such as MonoLayout and OFT seem to report the mAP (mean average precision) metric in addition to the mIoU metric, because mAP often turns out to be a more accurate estimate of prediction performance (due to integrating over various recall values). In practice, this leads to less-than-perfect predictions being scored well (and this could explain the splayed-out results in Fig. 2 scoring a high mIoU). Evaluating mAP would be a stricter criteria, and will allow an additional point of comparison with prior art.\n\n\n## Minor remarks\n\nThe following remarks have had no impact on my assessment of the paper, and as such I don't expect the authors to respond to these.\n\nConcurrent approaches such as [F] can be cited and discussed.\n\nThe paper could be structured better. For instance, input image sizes and baselines could be moved over to the main paper, rather than being listed in the appendix.\n\n## References\n\n[A] Mani, Kaustubh, et al. \"MonoLayout: Amodal scene layout from a single image.\" The IEEE Winter Conference on Applications of Computer Vision. 2020.\n\n[B] Lu, Chenyang, Marinus Jacobus Gerardus van de Molengraft, and Gijs Dubbelman. \"Monocular semantic occupancy grid mapping with convolutional variational encoder–decoder networks.\" IEEE Robotics and Automation Letters 4.2 (2019): 445-452.\n\n[C] Wang, Yan, et al. \"Pseudo-lidar from visual depth estimation: Bridging the gap in 3d object detection for autonomous driving.\" Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2019.\n\n[D] Roddick, Thomas, Alex Kendall, and Roberto Cipolla. \"Orthographic feature transform for monocular 3d object detection.\" arXiv preprint arXiv:1811.08188 (2018).\n\n[E] A Parametric Top-View Representation of Complex Road Scenes. CVPR 2019.\n\n[F] Lift, Splat, Shoot: Encoding Images From Arbitrary Camera Rigs by Implicitly Unprojecting to 3D. ECCV 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "b0OVQgJaUM5", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Interesting problem but the paper can be improved\n\nThis work aims to directly estimate the world layout in front of the vehicle from a pair of stereo cameras. It is based on cost volume but it does not explicitly predict the depth values of each pixel. Instead, it warps the cost volume features to the bird eye view (BEV) and do semantic segmentation from BEV using U-Net.\n\nI think the problem is interesting and I believe it has never been mentioned and/or addressed before. Moreover I think the motivation is also valid, as the BEV semantic segmentation from camera sensors can be one important perception input for navigation and planning. \n\nI like the idea of skipping the explicit 3D reconstruction and directly shoot for the final goal; I believe we usually get better performance when we directly minimize the loss we want to minimize. Moreover, it could potentially introduce some inspiration to other works, e.g., the direct extension - 3D (point/volume) semantic segmentation.\n\nThough I like this work, I also has several concerns:\n1. Is IPM feature really important? I only see it is effective on the synthetic dataset CARLA but not on KITTI. What is the possible reason? My guess is that the ground estimation is very bad for the real-world data. I am also curious what is the performance if only IPM feature is used. \n2. In introduction, this paper claims estimating accurate depth is not sufficient due to occlusion. However, I don't see how this work could handle occlusion. Instead the occluded part is masked out during training. Please explain this statement.\n3. What is the range of the layout estimation? From CARLA, it is 39m, and from Figure 5 and Figure 6, it is 35m. If it is the case, the short range of the estimation makes it hard to act as a major component in the perception system; the best use case is for short range detection and system redundancy. But actually I can imagine that it would not get very good result in long range, as there is always a trade-off between baseline (for accuracy) and camera overlap (for coverage) in stereo estimation.\n4. What is the image resolution for the inference time test? It seems quite slow if the resolution is 512x288 (for CARLA) or 640x256 (for KITTI).\n5. For the experiment, I think it is better to report mean $\\pm$ std with multiple trainings, as there are training noise.\n\nOther suggestions and clarifications\n1. When IPM is first introduced in Page 2, it is better to explain it in a short sentence. The current version is not clear and there are typos.\n2. I believe this work is based on binocular stereo pairs (correct me if I am wrong), so please explicitly say that in the paper. Also, using left/right image instead of reference/target image is less misleading.\n3. For disparity feature volume, it is better to use the prevalent name - cost volume. It is called cost volume in the introduction but later called disparity feature volume, I think it is better to be consistent.\n4. It is unclear how IPM feature are obtained: from pre-determined parameters or ground estimation? I think pre-determined parameters will not work very well because ground is not always a perfect plane.\n5. It is unclear what is the ensemble method used here. If it just takes the best of several models, I will not be convinced.\n\nAfter rebuttal:\nI still think this work has a interesting task setup, though it indeed has many faults (after reading the responses and other reviewers):\n1. It seems that IPM is not really useful in practice.\n2. It is also not sufficient to large occlusion, and thus there is no explanation for its advatange over `estimating accurate depth`\n3. Range is short and latency is high\n4. After reading reviewer1's comments, I think it could use the same experimental setting as the existing methods for a fair comparison. The other methods might be not properly trained with the new setting.\n5. It is still not clear how to emsemble several models (with different trained weights) in this work.\nThus I am changin my rating to 6, and I will not fight for this work.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HYETgkIjh0u", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposed to estimate the semantic layout in the bird eye's view from a pair of stereo images. The main novelty/contribution lies in how to organize and exploit the information from the stereo images. The proposed framework builds upon inverse perspective mapping, and projected stereo feature volume. The performance was evaluated on the KITTI and CARLA datasets. Given a pair of stereo images, there are various options to exploit the image information, where this paper provides a framework by exploiting the stereo information in the bird eye's view. \n\n- A principled question is what is the real superiority of estimating the layout in the bird eye's view. From the application' view, the semantic estimation from the camera'view already provide much information which the stereo images could further improve the performance. From the applications's perspective, I would like to see discussions and experiments in showing the superiority in using the bird eye's representation. \n\n- In the ablation studies, the paper already provide different variants of the network architecture in exploiting the stereo image information. I believe there are multi-task learning based framework form this task, where the semantic layout estimation and stereo estimation are jointly estimated and optimized. Whether that pipeline will provide extra benefit?\n\n- In section 3.4.4, the paper claimed that \"We pass the concatenated stereo BEV feature map and IPM BEV feature map to a U-Net (Ronneberger et al., 2015) network to generate the semantic map C\". However, the loss evaluation applies to the IPM features and Stereo features separately, namely, $\\mathcal{C}_i^{IPM}$ and $\\mathcal{C}_i^{Stereo}$. If two estimations are made as the network output, which one will be used for performance evaluation? The other following question is : If two separate estimations are made as the network outputs and compared with the ground truth for loss evaluation, whether a consistency loss between these two estimations will further constrain the network learning?\n\n- The paper conducted experiments on the KITTI and CARLA dataset. It is well understood that the CityScape dataset has been widely in evaluating semantic segmentation where the stereo images are available. I would to see more evaluation on these real-image dataset rather than synthetic dataset such as the CARLA dataset.\n\n- The paper title and abstract should highlight \"semantic\" and \"bird eye's view\" as the paper proposed to learn the semantic layout in the bird eye's view. The current title did reflect these properties.\n\nAll in all, taking all the above comments into consideration, I would like to hear from the authors' response, which could lead to updated rating in either directions.", "extra_scores": {}, "extra_text": {}}, {"review_id": "8mo0cvbXIpn", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an end-to-end network for layout estimation from stereo images. The approach is built off previous stereo matching networks, which built and process a 3D disparity volume. The stereo estimate is used to project image features into a birds-eye-view representation which is processed using a U-net which predicts a semantic scene layout. The approach is evaluated on the KITTI and Carla generated datasets.\n\nStrengths:\n* This is the first work to attempt semantic layout estimation from stereo images\n* The approach is geometrically grounded, and can properly leverage stereo information to improve layout estimation\n* The approach performs well on the two datasets evaluated. Since this paper is focused on a new problem, there are not existing works to directly compare to. However, the paper provides reasonable baselines by modifying existing networks for this task\n* Avoids the need for an intermediate representation (i.e. point cloud) by directly mapping features from the disparity volume into birds-eye-view coordinates\n* Plots in the appendix are interesting\n\nWeaknesses:\n* While the task itself is new, closely related forms of the problem have been studied. For example 3D object detection from monocular/stereo, and monocular layout estimation. It would have been helpful to see results on the closely related task of 3D object detection to better compare against prior works.\n*  The IPM module appears to be very sensitive to the accuracy of the ground plane. In the synthetic CARLA dataset, where a ground plane can be accurately computed, there seems to be a large advantage of using the IPM module. On real-world data like KITTI, the use of the IPM module gives very limited improvement in performance of the stereo-only baseline.\n* The task is closely related to 3D object detection which has been using similar components. The core components of the approach have been used in various forms in prior work. The paper (Orthographic feature transform for monocular 3d object detection, Roddick 2019) uses a very similar method to project image features into a birds-eye-view representation.\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper addresses the problem of estimating a “birds-eyed-view” overhead semantic layout estimate of a scene given an input pair of stereo images of the scene. The authors present an end-to-end trainable deep network that fuses features derived from the stereo images and projects these features into an overhead coordinate frame which is passed through a U-Net style model to generate the final top view semantic segmentation map. The model is trained in a fully supervised manner. Experiments are performed on the CARLA and KITTI datasets. 

While R2 was positive, they still had some concerns after reading the rebuttal and the other reviews. Specifically, they were not convinced about the value of the IPM module. This concern was also shared by R4, especially in light of the relationship to Roddick et al. BMVC 2019. R1 had concerns about the experiments, specifically the quantitative comparisons to MonoLayout. The authors addressed these comments, but it is still not clear if the differences can be attributed to the number of classes, how they are weighted, or the training split used? R3 had questions about the utility of BEV predictions in general. However, as stated by R2, there is a lot of value in approaching the problem in this way. 

In conclusion, while there were some positive comments from the reviewers, there were also several significant concerns. With no reviewer willing to champion the paper, there is not enough support to justify accepting the paper in its current form. 

Decision: Reject