Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation
Authors: Zhi Liu
Shaoyu Chen
Xiaojie Guo
Xinggang Wang
Tianheng Cheng
Hongmei Zhu
Qian Zhang
Wenyu Liu
Yi Zhang
Venue: CoRL 2022 Poster (2022)
Keywords: Polar Rasterization
Surface Estimation
BEV Segmentation

Abstract: In this work, we propose PolarBEV  for vision-based uneven BEV representation learning. To adapt to the foreshortening effect of camera imaging, we rasterize the BEV space both angularly and radially, and introduce polar embedding decomposition to model the associations among polar grids.  Polar grids are rearranged to an array-like regular representation for efficient processing. Besides, to determine the 2D-to-3D correspondence, we iteratively update the BEV surface based on a hypothetical plane, and adopt height-based  feature transformation. PolarBEV keeps real-time inference speed on a single 2080Ti GPU, and outperforms other methods for both BEV semantic segmentation and BEV instance segmentation. Thorough ablations  are presented to validate the design. The code will be released for facilitating further research.

Vision-based Uneven BEV Representation Learning with Polar Rasterization and Surface Estimation
Zhi Liu 13 Tianheng Cheng 2.3
Shaoyu Chen 2.3 x
Xiaojie Guo if
Xinggang Wang Wenyu Liu
Hongmei Zhu
Qian Zhang
Yi Zhang
Tianjin University Huazhong University of Science & Technology Horizon Robotics
Abstract: In this work, we propose PolarBEV for vision-based uneven BEV representation learning. To adapt to the foreshortening effect of camera imag- ing, we rasterize the BEV space both angularly and radially, and introduce polar embedding decomposition to model the associations among polar grids. Polar grids are rearranged an array-like regular representation for efficient process- ing. To determine the 2D-to-3D correspondence, we iteratively update the BEV surface based on a hypothetical plane, and adopt height-based feature transfor- mation. PolarBEV keeps real-time inference speed on single 2080Ti GPU, and outperforms other methods for both BEV semantic segmentation and BEV instance segmentation. Thorough ablations presented to validate the de- sign. The code has been released for facilitating further research at https: //github. com Stom//Spper-Li/PooaaBBWW
Keywords: Polar Rasterization, Iterative Surface Estimation, BEV Segmentation
Introduction
Bird's Eye View (BEV) representation [1, 2, 3, 4, 5, 6. 7] is of great practical value for environ- mental perception in autonomous driving. Especially for vision-based system, BEV implicitly and elegantly aggregates multi-view information into unified representation, avoiding time-consuming post processing for multi-vier fusion.
This work proposes PolarBEV for iusion-based uneven BEV representation learning, We rasterize the BEV space both angularly and radially, making BEV grids densely distributed near the ego- vehicle and sparsely distributed far from the ego-vehicle, i.e., distance-dependen uneven grid dis- tribution. Recent works [2, 9] rasterize the BEV space along the cartesian axes and get evenly distributed rectangular grids. Such rectangular BEV representation is straightforward, but polar BEV representation makes more sense. First, for self-driving car, the concerned perception region centered at the ego vehicle. Surrounding perception results are more important than distant ones for avoiding traffic accidents. Thus, higher resolution in surrounding areas is expected. Second, for even BEV representation, long-range BEV space (e.g., 100m x 100m) requires large number of BEV grids and high computational budget. Polar rasterization enables long-tailed uneven grid distribution, which can be flexibly adjusted cover large BEV space with limited computation cost.
We assign angle-specific and radius-speciffs embeddings to each polar grid according its 3D position. Because of the foreshortening effects of camera imaging, object's scale in image varies lot when the distance to camera changes. Polar grids at the same distance correspond to the same scale. Grids at the same angle correspond to the same camera view. With angle-specific and radius- specific embeddings, we model the associations among grids to enhance the BEV representation.
We propose iterative surface estimation for effective and efficient BEV representation learning, Pre- vious methods [2, 9] usually predict pixel-wise depth distribution and broadcast pixel features to BEV space. Differently, we first set a hypothetical BEV surface and iteratively update the height of each polar grid adjust the 2D-to-3D correspondence between image pixels and BEV grids. Height
6th Conference on Robot Learning (CoRI. 2022), Auckland, New Zealand.

---

is much easier to be estimated than depth. The iterative refinement process leads to more precise 2D-to-3D feature transformation and better BEV representation.
PolarBEV achieves real-time inference speed (25 FPS on a 2080Ti GPU), and significantly outper- forms counterparts for both BEV semantic segmentation and BEV instance segmentation. Given the high efficiency and strong performance, PolarBEV can be integrated into autopilot system for online environmental perception.
Related Work
Recently, many large multi-sensor datasets [10, 11, 12] made it possible to directly supervise models by projecting 3D annotations onto the ground plane generate BEV labels. The key to the problem is how model the transformation from image view to Bird's Eye View, which is inherently an ill-posed problem. A straightforward method to assume the world is flat and transform the image to BEV map through Inverse Perspective Mapping (IPM) [1, 13, 14]. Though this approach works in some cases, it often introduces artifacts to objects that lie above the ground plane.
In order to achieve better results, other methods [15, 16, 17] explicitly estimate depth to lift objects into BEV. And OFT [18] maps image-based features into an orthographic 3D space with the aid of camera parameters. A potential performance bottleneck of this method is that the contribution of each pixel feature is independent of objects depth that pixel. Instead of copying each pixel feature along camera ray, Lift-Splat [9] learns depth distribution for each pixel. Recently, FIERY [2] has extended Lift-Splat [9] further to use multi-timestamp observations for motion forecasting. Different from these methods, we adopt BEV surface estimation instead of depth distribution to determine the correspondence between image and BEV.
Another technical route directly predicts BEV outputs from input images. CVT [8] encodes the camera parameters into positional embeddings to model the geometric structure of the scene implic- itly. Different from CVT [8] using global attention to update each query, GKT [7] leverages the geometric priors to guide the transformer to focus on discriminative regions. VED [6] predicts a semantic occupancy grid directly from the front-view image with variational encoder-decoder net- work. VPN [19] proposes a fully-connected view relation module to predict the semantic BEV map from multiple views. PON [20] further advances fully-connected layer for each column to translate features from image space to BEV space. Instead of using fully connected layers, TIM [21] models the relation of image columns and BEV polar rays with eross-attention. Based TIM [21], the work [22] further employs a graph network to spatially reason about an object within the context of other objects. BEVFormer [23] predefines a set of uniformly distributed height anchors and projects these anchor points to image to get features.
In 3D domain, [24, 25] divide the 3D space into polar grids for point cloud segmentation, in order to adapt to the long-tailed distribution of LiDAR points. [26] leverages the radial symmetry to nor- malize point cloud along the radial direction. [27] introduces polar parametrization for 3D detection to establish explicit associations between image patterns and prediction targets. Differently, consid- ering the foreshortening effects of camera imaging and the characteristics of BEV perception, we adopt polar rasterization for vision-based BEV representation learning.
3 Method
3.1
Overview
The framework of PolarBEV is presented in Fig. 1. Taking multi-view images as input, we first extract image features with shared CNN backbone. We rasterize the BEV space along the polar coordinates and rearrange the polar grids to array-like regular representation. Then, we iteratively update the BEV surface with grid-wise height estimation, and transform features to BEV features based on the estimated height and camera's 's calibrated parameters. BEV surface updating and feature transformation are repeated a cascade manner for several times. And various heads follow the final BEV representation to perform BEV perception. Detailed designs are presented below.

---

Circular Padding
Height- beaigtt-based based Projection
Grid B
Rmax
GridA
Rearrange
Drad
Dang
Surround- View Image Featuros
Polar BEV Rasterization
A Arry--like like Polar Representation
Camera
Height- based Projection
Updated BEV Surface
Image Plane
AhA
A/pp
Hypothetical BEV Surface
Grid
Grid B
Illustration of PolarBEV. Top: multi-view images are sent into the shared CNN back- bone for feature extraction. BEV space is rasterized along the polar coordinates. Polar girds are rearranged to array-like regular representation. Bottom: we iteratively update the BEV surface for precise height- height-based based projection.
Figure 1:
3.2 Polar BEV Rasterization and Rearrangement
The concerned BEV space is centered the ego-vehicle with the radius Rmaz and 360 FoV (field of view) coverage. As shown in Fig. 1, we rasterize the BEV space both angularly and radially. Ra- dially, evenly divide (0, Rmazz] into Drad segments (Rmaz and Drad respectively denote radius maximum and radial resolution). Angularly, we evenly divide the 360 into Dang segments (Dang denotes the angular resolution). After rasterization, for efficiently processing the polar representa- tion, we rearrange polar grids along the angular and radial dimensions and get array-like regular representation with shape Drad Dang- The rearranged representation is hardware-friend friendly but can not be processed by conventional convolution operation. As shown in Fig. 1, in the angular dimension, - and π correspond the the same angle but are separated in the array-like regular representation. Alternatively, we adopt ring convolution [25] with circular padding to process the rearranged representation. Specifically, we first circularly pad in the angular dimension, and then adopt conventional convolution with 0 padding in the radial dimension. The ring convolution is followed by batch normalization and ReLU activation layers.
3.3 Polar Embedding Decomposition
For each polar grid with radial distance r and angle 0, we predefine learnable query embedding q and decompose q into two components, i.e., radius-specific query ( and angle-specific query (0)
gang, which is formulated
(0) grad + Lang
(1)
For the polar representation, polar grids at the same distance correspond to the same scale, and grids at the same angle correspond the same camera view. With angle-specific and radius-specific embeddings, we model the associations among grids to enhance the BEV representation. Ablation experiments are presented in Sec. 4.3 validate the effectiveness of embedding decomposition.
3.4 Iterative Surface Estimation and 2D-to-3D Feature Transformation
We perform 2D-to-3D feature transformation based on height-based projection mechanism. To make sure the correspondence between image and BEV, we first set hypothetical BEV surface with height haypo and then iteratively update the height of each grid i based on its query embedding

---

qs. This can be formulated as follows:
(2) where is MLP layer, is the iteration index and hy the hypothetical height hypo Then we normalize hy (superscript is omitted for clarity) to the range of [0, 1] with sigmoid function and further scale up the value to the range of [Zmn Zeupl. i.e.,
b (á + +
E; = (h; x (Zsup- - Zinf) Zinf=
(3)
where Zaup and Z the predefined upper and lower bounds of height respectively, Each grid i corresponds to polar coordinate p: (r:. 02) (the position of the grid's center point). We first
transform each polar coordinate p; to cartesian coordinate (æg, y4) in the following manner.
24 r; cos 1/1 P2 X sin 0
Then we concatenate the cartesian coordinate (2i, U1) with its corresponding height z; to construct homogeneous coordinate W/, i.e.,
w3 (a,, v1) 2
(5)
where denotes the concatenation operation, Then we project we the image plane with camera's intrinsics / E R x3x3 and extrinsics E E x3x4.
Us,n = In En W3
(6)
where Mj,n the projected image point on the nth view of BEV grid i, and represents the total image views. Finally, we transform features from the image view to the Bird's Eye View with the projected coordinates. This can be formulated as:
K -= D=ΣM F
12
where Fin is the sampled image feature corresponding to the projected image point uin. and M,, is binary mask for masking out projected points which exceed the image boundary. After that, fa sent into a MLP layer and its output added to the previous query for query update.
The surface estimation and 2D-to-3D feature transformation are conducted for several times and the polar representation is iteratively updated.
3.5 Segmentation Head
We follow the head design of FIERY [2]. We adopt a small encoder-decoder network to further refine BEV features. Three branches follow the network, respectively for predicting segmentation score, offset and centerness. The loss settings are also the same with FIERY [2]. Polar predictions mapped into rectangular predictions for loss calculation based the rectangular ground truths.
4 Experiments
4.1 Experimental Settings
NuScenes [10] dataset is a large-scale autonomous driving dataset which was collected over a variety of weathers and traffic conditions. This dataset contains 1000 scenes and each scene lasts 20 seconds. The captured RGB images are from six cameras covering a full of 360 around the ego-vehicle. Each camera has calibrated intrinsics and extrinsics at every timestamp. Following common practice [2, 8), we project the 3D box annotations of vehicles onto the ground plane to get ground truth labels.
Datasets
Architecture We adopt the pre-trained EfficientNet-B4 [28] as backbone to extract multi-view image features. Features of stride are taken as the input for feature transformation. The initial polar BEV feature is combination of angular queries and radial queries with the shape of 400 100 64 for representing certain BEV area (setting 1 or setting 2) centered at the ego-vehicle. We set the radius maximum Rman as the distance from the farthest point of BEV area to the center of ego- vehicle. The small encoder-decoder network contains first four layers of ResNet-18 [29], and all the convolution layers with kernel size greater than 1 are changed into ring convolutions to adapt to the polar representation.

---

Method
Setting ((loU%)
Setting 2(1oU%)
# Params(M)
FPS
PON [20]
24.7
VPN [19]
25.5
STA [33]
36.0
Lift-Splat [9]
32.1
14
25
FIERY Static [2]
37.7
35.8
7.4
Ours 4824x480)
41.5
37.6
7.4
25
Ours (448 x60)
45.4
41.2
7.4
10
Comparison of vehicle BEV semantic segmentation nuScenes [10] without masking invisible vehicles. Setting 1 refers to 100m x 50m at 25cm resolution. Setting 2 refers to 100m x 100m at 50cm resolution. We report iniersection-Ove--nnii BEV semantic segmentation performance. The iteration of feature transformation set to for both input resolution, (224 480) and (448x 960) in this table. Our method (224x480 input) significantly outperforms counterparts while achieving real-time speed (25 FPS).
Table 1:
(IoU) for evaluating vehicle
Method
Setting 1(loU%)
Setting 2(IoU%)
# Params(M)
FPS
FIERY Static [2]
42.7
39.8
7.4
CVT [8]
37.5
36.0
35
Ours (224 480)
44.3
41.3
7.4
25
Ours (448 9600
48.4
45.6
7.4
10
Table 2: Comparison of vehicle BEV semantic segmentation nuScenes [10) with masking invisi- ble vehicles. The iteration of feature transformation is set to 2 for both input resolution, (224 x 480) and (448 960) in this table. Our method (224 x 480 input) achieves much higher IoU than FIERY Static [2] and CVT [8] while achieving real-time speed (25 FPS).
Training All our networks are implemented with Pytorch [30] and Pytorch Lightning. The input images first resized and randomly cropped into special resolution before being fed into the networks. The AdamW [31] optimizer used for training the networks, with weight decay of le-7 We apply the one-cycle learning rate scheduler [32] to adjust the learning rate. All networks trained on 8 GPUs with a batch size of 16.
Evaluation Two settings are used for evaluating vehicle segmentation map. Setting 1 [20] per- ceives a 100m x 50m area around the ego-vehicle with 0.25m resolution, while the perceptual range of setting [9] 100m 100m with 0.5mm resolution. These two settings serve as the main comparisons to prior works. We use setting 2 for all the ablation studies. The Intersection- Over-- Union (IoU) metric is used for evaluating vehicle BEV segmentation performance for both settings. We also report Panoptic Quality (PQ) metric for evaluating vehicle BEV instance segmentation per- formance, following previous work [2]. Additionally, we report inference speeds measured on single 2080Ti GPU. The number of iterations and input resolution for PolarBEV are set to 3 and 224 480 respectively for most experiments unless specified otherwise. Results of ablation studies produced with masking invisible vehicles.
4.2 Performance Comparison
To validate the effectiveness and efficiency of the proposed method, we compare PolarBEV with competitive approaches both BEV semantic segmentation and BEV instance segmentation. All the methods involved in comparison only use single frames as input without temporal information.
BEV Semantic Segmentation For fair comparison, in Tab. and Tab. 2, we respectively compare results without and with masking invisible vehicles. Tab. I compares results without masking invis- ible vehicles. Our model with 2 iterations outperforms counterparts by significant margin both settings, especially for 448 x 960 input resolution. Even compared with the best prior method FIERY Static [2), our method (224 x 480 input) is still 3.8 points higher in setting 1 and performs faster inference with the same parameters. When the input resolution is increased to 448 x 960, our model
5

---

Setting SQ%
Setting
FPS
RQ%
PQ%
RQ%
SQ %
PQ%
FIERY Static [2]
49.5
71.7
35.5
47.3
71.1
33.6
Ours
52.3
72.0
37.7
50.2
71.4
35.9
25
Table Comparison of vehicle BEV instance segmentation on nuScenes [10] with masking invisi- ble vehicles. We report Recognition Quality (RQ), Segmentation Quality (SQ) and Panoptic Quality
(PQ) for evaluating vehicle BEV instance segmentation performance. Our method significantly out- performs FIERY Static [2] in both performance and speed.
Grid Distribution
IoU%
PQ%
FPS
Feature Transformation
IoU%
PQ%
FPS
Rectangular
40.83
35.45
22
Depth-based
39.81
33.62
8
Polar
41.36
35.78
22
Height-based
40.83
35.45
22
Table 4: Ablation study about polar and rectan-
Table 5: Ablation study about depth and
gular grid distribution.
height-based feature transformation.
still runs in a faster speed than FIERY Static. In Tab. 2, we compare PolarBEV with other methods with masking invisible vehicles. Because there no official results FIERY Static [2] in this setting, we reproduce it with the official code. As this table shows, model (224 x 480 input) with 2 iterations also achieves the best results in both settings and achieves real-time performance.
achieves further 7.7 points higher in setting
BEV Instance Segmentation We also provide BEV instance segmentation comparison with masking invisible vehicles in Tab. 3. The results of FIERY Static [2] are reproduced with the official code. As shown in this table, our model with 2 iterations more than two points higher than FIERY Static in both settings in terms of PQ. We also report Recognition Quality (RQ) and Segmentation Quality (SQ) in this table. As we can see from this table, the revenue mainly comes from RQ. meaning model detects instances more accurately.
4.3 Ablation Study
Polar vs. Rectangular Grid Distribution In Tab. 4, we present ablation experiments about polar and rectangular grid distribution. For fair comparison, both results have the same number of BEV grids, 200 x 200 for rectangular grid distribution and 400 100 for polar grid distribution. We keep the network structure the same. As shown in Tab. 4, polar grid distribution achieves an improvement of 0.53 IoU and 0.33 PQ with no degradation in inference speed.
Height-based vs. Depth-based Feature Transformation In Tab. 5, we provide ablation experi- ments about feature transformation. FIERY Static [2] adopts depth-based transformation, thus we directly take it as baseline for comparison. To get the height- baaeed results, based on FIERY Static, we change the feature transformation manner to the one introduced in Sec 3.4. Tab. shows that height-based results surpass depth-basee based results lot in both ToU and PQ with nearly 3x inference speed, validating the advantage of height-based transformation.
The advantage mainly comes from the following points. The numerical range of depth is |0, +oo) for each pixel, which is hard for the network estimate reasonable value in such a huge space. While
Ring Convolution
PED (Jangs Trad)
loU%
PQ%
40.48
34.39
40.81
35.42
41.36
35.78
Table 6: Ablation study about ring convolution and embedding decomposition. PED denotes Polar Embedding Decomposition.

---

Angular
Radial
I00%
PQ%
Params(M)
50
100
37.43
26.63
7.5
100
100
39.53
31.77
7.5
200
100
40.46
34.63
7.5
400
100
41.36
35.78
7.5
600
100
41.40
35.93
7.6
400
50
40.34
34.60
7.5
400
142
41.48
36.13
7.5
Table 7: Ablation study about various resolutions of polar BEV representation. The shape of polar BEV representation is Dang Drad-
I0U%
PQ%
#Params(M)
FPS
40.73
34.59
7.3
27
41.30
35.86
7.4
25
41.36
35.78
7.5
22
41.39
35.96
7.8
17
Table 8: Ablation study about iterations of the 2D-to-3D feature transformation.
the numerical range of height for BEV is quite small. Height-based transformation corresponds to less prediction error. As for the speed, depth-based methods (such as FIERY Static) broadcast pixel features BEV space along the depth dimension and sum the 3D features along the vertical di- mension. These operations are time-con time-conuumin. While height- based transformation is lightweight, resulting in less latency.
Ring Convolution and Polar Embedding Decomposition In Tab. 6, we ablate on the ring convo- lution and the proposed polar embedding decomposition. The baseline result achieves 40.48% loU and 34.39% PQ. After adopting ring convolution, we observe an improvement of 0.33 in IoU and 1.03 in PQ respectively. In order model the associations among polar grids, we decompose grid embeddings into angle-specific ones gang and radius-specific ones grod- As the last row shows, embedding decomposition further brings an improvement of 0.55 IoU and 0.36 PQ.
Polar BEV Resolution To verify how the resolution of polar BEV representation affects model performance, we performe ablation experiments on the angular resolution and the radial resolution respectively. As shown in the upper of Tab. 7, the model performance becomes better with the increase of angular resolution and reaches saturation at 400. The same rule can found in radial resolution, as the radial resolution increases, the results become better. Although increasing the angular resolution or the radial resolution brings benefits and barely increases model parameters, the memory occupation is proportional to the angular or radial resolution. Considering both the performance and memory occupation, we choose the resolution of 400 x 100 as the default setting.
The Number of Iteration We also conduct ablation study about the number of BEV surface timation iterations in Tab. As this table shows, our model with only iteration gets 73% IoU and 34.59% PQ, which has already exceeded all previous methods by significant margin but with a real-time inference speed. When the number of iteration is 2, the method achieves higher per- formance in both IoU and PQ metrics and still keeps real-time speed. After adding the number of iteration to 3 or 6, the performance seems to reach saturation, while the FPS degrades a lot when the number of iterations is 6.
4.4 Qualitative Results
Fig. shows several qualitative results on various scenes. give the six camera views and the predicted segmentation results and instance segmentation results along with the ground truth in each row. As can be seen from this figure, PolarBEV can accurately segment vehicles in a variety of complex scenes.
7

---

FRONT
RICKY SOG
BAR
BXX
RAC SIGHT
TRGHELETT
TACNT
TRONT SIGHT
RaCC LEFT
BACK
BACK RIERTT
Figure 2: Qualitative results on various scenes. Left shows the six camera views. Right shows our semantic segmentation results, instance segmentation results and the ground truth in turn.
TRONT
Figure 3: Failure examples on nuScenes [10]. Left shows the six camera views. Right shows our semantic segmentation results, instance segmentation results and the ground truth in turn.
4.5 Limitations
We present some failure examples in Fig. 3. As shown in the upper of this figure, it is hard for PolarBEV to predict the exact location when vehicle far away from the ego-vehicle. bottom of this figure shows, PolarBEV fails when the height of vehicle is hard to estimate because of blocking. In order achieve better results in the future, we may consider using temporal information to calibrate the failures.
the
5 Conclusion
We propose PolarBEV for vision-based uneven BEV representation learning in this work. PolarBEV rasterizes the BEV space both angularly and radially, making distance-dependen uneven grid distribution. To model the associations among grids, we assign angle-specific and radius-specific embeddings to each polar grid. Different from previous depth based methods, first set hypo- thetical BEV surface and then iteratively update the height each polar grid to adjust the 2D-to-3ID correspondence between image pixels and BEV grids. Extensive experiments show PolarBEV is alternative way for better and faster segmentation.

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grant no. 62072327.
References
[1] S. Ammar Abbas and A. Zisserman. A geometric approach to obtain bird's eye view from an image. In Proceedings of the IEEF/CVF International Conference on Computer Vision Workshops, 2019.
[2] A. Hu, Z. Murez, N. Mohan, S. Dudas, J. Hawke, Badrinarayanan, R. Cipolla, and A. Kendall. Fiery: Future instance prediction in bird's-eye view from surround monocular cameras. In ICCV, 2021.
[3] Kanade and M. Okutomi. A stereo matching algorithm with an adaptive window: Theory and experiment. IEEE transactions on pattern analysis and machine intelligence, 1994.
[4] Y. Kim and D. Kum. Deep learning based vehicle position and orientation estimation via inverse perspective mapping image. In 2019 IEEE Intelligent Vehicles Symposium (IV), 2019.
[5] A. Loukkal, Grandvalet, T. Drummond, and Y. Li. Driving among flatmobiles: Bird-eye- view occupancy grids from monocular camera for holistic trajectory planning. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021.
[6] C. Lu, M. J. G. van de Molengraft, and G. Dubbelman. Monocular semantic occupancy grid mapping with convolutional variational encoder-decode networks. IEEE Robotics and Au- tomation Letters, 2019.
[7] S. Chen, T. Cheng, X. Wang, W. Meng, Q. Zhang, and W. Liu. Efficient and robust 2d-to-bev representation learning via geometry-guides kernel transformer. arXiv preprint 20Xi::200600584, 2022.
[8] B. Zhou and P. Krähenbühl. Cross-view transformers for real-time map-view semantic seg- mentation. arXiv preprint arXiv:2205.02833; 2022.
[9] J. Philion and S. Fidler. Lift, splat, shoot: Encoding images from arbitrary camera rigs by implicitly unprojecting to 3d. In ECCV, 2020.
[10] H. Caesar, V. Bankiti, A. H. Lang, S. Vora, V. Liong, Q. Xu, A. Krishnan, Y. Pan, G. Baldan, and O. Beijbom. nuscenes: multimodal dataset for autonomous driving. CVPR, 2020.
[11] A. Geiger, P. Lenz, and R. Urtasun. Are ready for autonomous driving? the kitti vision benchmark suite. In 20/2 IEEE conference on computer vision and pattern recognition, 2012.
[12] Sun, H. Kretzschmar, X. Dotiwalla, A. Chouard, V. Patnaik, P. Tsui, J. Guo, Y. Zhou, Chai, B. Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
[13] L. Reiher, B. Lampe, and L. Eckstein. A sim2real deep learning approach for the transfor- mation images from multiple vehicle-mounted cameras to a semantically segmented image in bird's eye view. In 2020 IEEE 23rd International Conference Intelligent Transportation Systems (ITSC), 2020.
[14] Y. B. Can, A. Liniger, D. P. Paudel, and L. Van Gool. Structured "ed''s-eye-veww traffic scene understanding from onboard images. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, 2021.
[15] Z. Wang, B. Liu, S. Schulter, and M. Chandraker. A parametric top-view representation of complex road scenes. Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, 2019.

---

[16] S. Schulter, M. Zhai, N. Jacobs, and M. Chandraker. Learning to look around objects for top-view representations of outdoor scenes. In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
[17] B. Liu, B. Zhuang, S. Schulter, P. Ji, and M. Chandraker. Understanding road layout from videos as a whole. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020.
[18] T. Roddick, A. Kendall, and R. Cipolla. Orthographic feature transform for monocular 3d object detection. arXiv preprint
2018.
[19] B. Pan, J. Sun, H. Y. T. Leung, A. Andonian, and B. Zhou. Cross-view semantic segmentation for sensing surroundings. IEEE Robotics and Automation Letters, 2020.
[20] Roddick and Cipolla. Predicting semantic map representations from images using pyra- mid occupancy networks. In CVPR, 2020.
[21] A. Saha, O. M. Maldonado, C. Russell, and R. Bowden. Translating images into maps. arXiv preprint 1rXiv:2100000666 2021.
[22] A. Saha, O. Mendez, C. Russell, and R. Bowden. " the pedestrian next to the lamppost" adaptive object graphs for better instantaneous mapping. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19528-19537, 2022.
[23] Z. Li, W. Wang, H. Li, E. Xie, C. Sima, T. Lu, Y. Qiao, and J. Dai. Bevformer: Learning bird's- eye-view representation from multi-camera images via spatiotemporal transformers. arXiv preprint arXiv:2203.172700 2022.
[24] H. Zhou, X. Zhu, X. Song, Y. Ma, Z. Wang, H. Li, and D. Lin. Cylinder3d: An effective 3d framework for driving-scene lidar semantic segmentation. arXiv preprint arXiv:2008.013: 015500 2020.
[25] Y. Zhang, Z. Zhou, P. David, X. Yue, Z. Xi, B. Gong, and H. Foroosh. Polarett An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2020.
[26] S. Chen, X. Wang, T. Cheng, W. Zhang, Q Zhang, C. Huang, and W. Liu. Azinorm: Exploiting the radial symmetry of point cloud for azimuth-now mornuut-normmlizedd 3d perception. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6387- 6396, June 2022.
[27] S. Chen, X. Wang, T. Cheng, Q. Zhang, C. Huang, and W. Liu. Polar parametrization for vision-based surround-view 3d detection. arXiv:2206. (00.2200..0065,, 2022.
[28] M. Tan and Q Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In ICML, 2019.
[29] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[30] A. Paszke, S. Gross, S. Chintala, G. Chanan, E. Yang. Z. DeVito, Z. Lin, A. Desmaison,
L. Antiga, and A. Lerer. Automatic differentiation in pytorch. 2017.
[31] Loshchilov and F. Hutter. Decoupled weight decay regularization. arXiv preprint arXiv: 1.051105100111 2017.
[32] N. Smith and N. Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for domain operations applications, 2019.
[33] A. Saha, O. Mendez, C. Russell, and R. Bowden. Enabling spatio-temporal aggregation in birds-eye-view vehicle estimation. In ICRA, 2021.
10


Reviews:
[{"review_id": "u_jG1KmPkX_", "reviewer": "Reviewer_iJAs", "summary": "The paper proposes a method for projecting features from 2D images into a polar bird eye's view grid. This grid is then used for estimating a 2.5D height surface of what is around the cameras. Features are then re-projected into an array-like polar representation and fed into an encoder-decoder based on FIERY [2].", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors propose to use polar rasterization for images inputs. This is a novel approach that changes how the inputs of a neural network can be encoded. The method is well evaluated and show some improvements when compared to the state of the art.\n\nThe clarity of the method section needs to be improved as some of the descriptions are confusing and could interpreted in different ways.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "I do think the paper proposes an interesting contribution with is novel and interesting. I do think the writing needs to be improved for a more clear, simpler, and more consistent methodology. That aside the experiments are pretty thorough with ablation studies of several of the parameters and comparison with state of the art.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Good: Minor revision needed to improve organization or clarity.", "potential_impact": "", "robotics_focus": "Sufficient demonstration on hardware", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "2: Unsure, difficult to assess.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "## Major comments\n* The notations for variables have several typos, which makes it confusing to follow, e.g., line 111 and 112, are {$z_{inf}, Z_{inf}$} and {$z_{sup}, Z_{sup}$} the same things? Same goes for line 100, and equation (1) with $p_{ang}$ and $q_{ang}$.\n* I do not understand how adding $q_{rad}$ and $q_{ang}$ allows forming a meaningful query system. How do you distinguish {$q_{ang} = 10^\\circ, q_{rad} = 1m$} from {$q_{ang} = 1^\\circ, q_{rad} = 10m$} (obviously things might be quantified differently but I think I missed something).\n* The methodology is hard to follow, I would encourage the author to clarify the distinction between the two types of projections (the first one with 3.2 and the second one in 3.4) by making the headings consistent with figure 1. Also, specify explicitly before equation (5) that you are doing the image re-projection.\n* Clarify what is the variable t, is equation (2) solved as a sliding window between frames, or is wrt to cells with closer radius, or is solved for each frame as an iterative optimisation starting from h_hypo until convergence?\n\n## Minor comments\n* line 150, do you mean 30k epochs? How do you track overfitting?\n* If setting 1 and 2 are used for evaluation, which settings do you use for training (Or how are the datasets divided between training and evaluation)?\n* Line 168, you compare with FIERY static, and not FIERY\n* Line 177, no need to redefine acronyms\n* Is using a constant resolution for $D_{rad}$ the best approach?"}}, {"review_id": "kBkkPOEApHM", "reviewer": "Reviewer_SUoV", "summary": "This paper tackles vision-based BEV semantic segmentation for on-road driving. It improves the previous work, FIERY, in two aspects: (i) using polar grid instead of a rectangular grid, and (ii) estimating the height of each map location instead of estimating the depth of each image pixel. Experiments validate the design choice and show that both the polar representation and the height estimation improve the BEV segmentation results on the nuscenes dataset.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- The proposed ideas are intuitive, effective and easy to implement.\n- The idea of iteratively estimating the height is interesting.\n- Experiments are thorough with detailed ablation study.\n\nWeaknesses:\n\nThe idea of using polar representation and height-based projection are not that new. There has been plenty of work of adopting polar representation for on-road driving (the authors have mentioned in the related work). Recently, there is also work that proposes height estimation for BEV segmentation [1]. While this work shows that the proposed methods are better than the baseline, I have several concerns of the proposed approach.\n\n[W1] Although the empirical results show that the polar representation is better, it is not clear why. Polar representation would have higher resolution close to the center, but its resolution at the edge of the map is lower than the rectangular representation. I think there is a tradeoff, and I wonder how the polar representation would perform as a function of the distance to the center. Is it always better? Or is it better near the center, and worse when far from the center than the rectangular representation? Does the dataset have a bias like this? It would be great if the authors could elaborate on this.\n\n[W2] While adding height estimation seems effective, it is unclear how well the height estimation works. Does it predict reasonable height values? Currently, it shows that the height estimation helps, but it is unclear if it is helping in the right way, i.e. predicting reasonable height values.\n\n[W3] The height estimation only estimates a single height value for each map cell. This discards a lot of the information from images. Does this have any potential drawbacks? A similar work [1] alleviates this issue by maintaining multiple height values in each map cell. It would be great if the authors could clarify the difference between this work and [1].\n\n[W4] The technical section could be made clearer. \n- In Sec 3.4, it is unclear how the sampled features $f_i$ be used to produce the query embedding $q$ in the next iteration.\n- I wonder if the comparison with previous work FIERY is fair. Are you using the same network architecture and parameter settings?\n\n[1] BEVFormer: Learning Bird's-Eye-View Representation from Multi-Camera Images via Spatiotemporal Transformers arXiv preprint arXiv:2203.17270\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "The paper contains some interesting ideas and is enjoyable to read. While the ideas are not entirely novel, they are applied to a slightly different task or in a different way, and detailed experiments have shown some advantages over existing methods. However, I would like to see some clarifications about where the performance gain comes from, and the relation to related work. It would be great if the authors could provide some additional details about the issues raised.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "issues": "* Please address the issues raised the previous section, namely i) why polar is better than rectangular and ii) the relation to existing work.\n* Make the technical section (3.4) clearer.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {}}, {"review_id": "6zvoxC8gW9_", "reviewer": "Reviewer_Zrsk", "summary": "This paper proposes a model that projects projective images from cameras mounted on a car to a polar birds-eye-view parametersization. The polar parameterization results in a non-uniform discretization of the space which has a higher resolution near the vehicle. The results show the parameterization achieves better segmentation accuracy compared to other methods such as FIERY and CVT. ", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n=========\n- As far as I’m aware this is the first method that uses a polar parameterization for a birds-eye-view representation of the area around a vehicle. This parameterization is ideally suited for this application as it greatly simplifies the processing and display of information.\n- The method is very simple and accounts for things like non-planar surfaces so it doesn’t make any assumptions about the curvature of the road. \n- The results show that the approach achieves better performance compared to other birds-eye-view approaches such as FIERY and CVT. \n\nWeaknesses\n=========\n- Apart from the detection of other car bounding boxes, there isn’t much explanation of what this parameterization and method can be used for. At the moment the functionality seems a bit limited. \n- How does the approach work for obstacles that don’t have standard configurations, such as boom barriers that don’t necessarily lie on supporting surfaces?\n- In terms of the evaluation, it’s not clear what contributes to the improvement in the accuracy in the segmentation task. This could either be the polar parameterization or the model itself.\n- There are quite a few grammar errors that need to be fixed. Eg. “besides” is repeatedly used in a strange manner. Line 128 starts with “And”.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "Overall, I think this type of representation is very suitable for autonomous vehicles and wheeled robots and the proposed method seems to perform well compared to existing methods. However, apart from the improved performance on bounding box estimation, it doesn't seem to show any impressive results or a significant advance in terms of theory, methods or model architectures.", "technical_quality": "Good: Generally convincing, but some additional experiments or analyses needed.", "clarity_of_presentation": "Very Good:  Well organized and clear.", "potential_impact": "", "robotics_focus": "Highly relevant to robotics but no hardware experiments", "quality_of_limitations": "Limitations are addressed clearly", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"impact": "3: The work is incremental and unlikely to have much impact, even though it may be technically correct and well executed.", "reviewer_expertise": "3: The reviewer is fairly confident that the evaluation is correct"}, "extra_text": {"issues": "- Clarify what parts of the method are leading to the improved segmentation accuracy compared to CVT and FIERY\n- Explain whether the method can be used for other tasks such as estimating the road curvature.\n- Fix the grammar errors\n"}}]

Author rebuttal:

Decision: Accept (poster)