Title: PolarStream: Streaming Object Detection and Segmentation with Polar Pillars
Authors: Qi Chen
Sourabh Vora
Oscar Beijbom
Venue: NeurIPS 2021
Keywords: Streaming Object Detection
3D Object Detection
LiDAR Segmentation
3D Panoptic Segmentation
Polar Grid

Abstract: Recent works recognized lidars as an inherently streaming data source and showed that the end-to-end latency of lidar perception models can be reduced significantly by operating on wedge-shaped point cloud sectors rather then the full point cloud. However, due to use of cartesian coordinate systems these methods  represent the sectors as rectangular regions, wasting memory and compute. In this work we propose using a polar coordinate system and make two key improvements on this design. First, we increase the spatial context by using multi-scale padding from neighboring sectors: preceding sector from the current scan and/or the following sector from the past scan. Second, we improve the core polar convolutional architecture by introducing feature undistortion and range stratified convolutions. Experimental results on the nuScenes dataset show significant improvements over other streaming based methods. We also achieve comparable results to existing non-streaming methods but with lower latencies.

TL;DR: This paper proposes a streaming model for simultaneous 3D object detection, Lidar segmentation and panoptic segmentation based on polar representation

Review discussion:
The main issue of this manuscript is lack of clarity and analysis which makes the claimed contribution unclear:
* In figure 1, the authors claim polar representation is better than cartesian in terms of memory and computation.
However, no such comparison is provided
* There is overall no result visualization or case analyses (segmentation+detection) to demonstrate the usefulness of the proposed method.
* For table 1, the detection result for <=4 sectors are worse than Cartesian, even with CP. Why is this the case? This seems to contradict with the claim that Polar representation is better. And why segmentation tasks do not have such an effect? Insights are needed.
* Range Stratified Convolution: this is extremely unclear for how the kernels are allocated for each grid.
Also relevant ablation studies are needed.
* Section 5.2, multi-scale padding: It is still unclear why detection performance improves when the number of sectors increases. It might be because there are more overlapping bbox proposals, or NMS strategy. More detailed analysis is needed.
In addition, only running the experiment on the NuScenes dataset is not sufficient in my opinion. I suggest the authors add the Waymo dataset.
Also directly accumulating 10 frames may incur localization error which may interfere with the detection result. Single frame baseline is also needed.
Some other issues:
* Not sure why the authors put emphasis on the point pillar backbone. Any other backbone can do the job.
* For feature undistortion, why only apply this method on the classification head? The more appropriate way is to apply on the backbone. This experiment is needed.
* Section 5.3, diagnosis part: this part of analyzing two previous methods is not relevant to the main body.
Evaluation
Originality: the method is original
Quality: the proposed method and experiments are technically sound.
Clarity: main issue is the clarity, mentioned in my comments
Significance: the low latency LiDAR detection model is of interest to broad audience in the community.
================post rebuttal====================
I would like to thank the authors for the detailed reply.
While the reply addressed some of my concerns, I am still not convinced by 1) the memory and computation advantage, and 2) the performance gain vs. the Cartesian coordinate (e.g., only larger sectors work, but sacrifices the overall performance).
Also the explanation of the solved cases and Range Stratified Convolution is still vague. My score remains the same.
No
We sincerely thank the reviewer for the constructive comments and hope our responses can address the concerns
- comparison between polar and cartesian in memory and computation
Cuboid-shaped voxels waste computation and memory because they use larger feature maps than ours. Feature map size comparsison is shown below. For more than 8 sectors, cartesian pillars use twice the feature maps size as ours because the way they partition the input region (Figure 4 showed an example of 8 sectors, where for cartesian pillars half of input region is empty).
| #sectors    | 1 | 2| 4| 8| 16| 32|
| ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |
| Cartesian    | 512x512       |512x256|512x128|512x128|512x64|512x32|
|Polar  | 512x512       |512x256|512x128|512x64|512x32|512x16|
Here is the memory usage of the feature map ‘canvas’ as referred to in PointPillars. The memory is per sector in MB:
| #sectors | 1 | 2| 4| 8| 16| 32|
| ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |
| Cartesian    | 33.6       |16.8|8.4|8.4|4.2|2.1|
|Polar  | 33.6      |16.8|8.4|4.2|2.1|1.3|
- visualization or case analyses
Our visualization shows that baseline methods have a lot of false positive detection bboxes at the boundaries for 32 sectors (because empty regions or ‘noises’ introduced by previous methods have similar effects to adversarial examples), while our polarstream with bidirectional padding have fewer false positives because we pad valid features. Here is an anonymous link for visualization: https://i.imgur.com/iYSmF2L.png
We will add visualizations to supplementary for revision.
- For table 1, the detection result for <=4 sectors are worse than Cartesian, even with CP. Why is this the case? This seems to contradict with the claim that Polar representation is better. And why segmentation tasks do not have such an effect? Insights are needed.
Our finding is cartesian representation is better for full-sweep detection, and polar representation is better for streaming and semantic segmentation. Detection results are worse for n <= 4 because full-sweep detection is worse and for n <= 4 sectors context padding does not show a lot of effects because the context is still enough (previous streaming methods do not have effects either). Streaming is an important onboard perception application for its reduced latency (in Figure 1 we reported 95ms for full-sweep and 14ms for 32 sector) and latency is extremely important because AVs must respond to the dynamic environment immediately. A polar coordinate system is ideally suited to streaming and enables context padding.
Why cartesian coordinates is better for full-sweep detection and polar coordinates is better for semantic segmentation? In the following table we show that cartesian coordinates has a higher performance upperbound in detection and polar coordinates has a higher upperbound in semantic segmentation.
| | pillar size | input size |det mAP upperbound|seg mIoU upperbound|
| -------- | -------- | -------- | -------- |-------- |
| Cartesian    | 0.2m x 0.2m     | 512x512 |98.9|92.4|
| Polar | 0.098m x 0.0123 rad| 512x512 | 96.7|95.1|
These upperbounds are what the models can get if learning is 100% correct. They are obtained by replacing predictions with ground truth labels during inference.
The upperbounds are not 100 for semantic segmentation because the network is doing pillar-level semantic segmentation and pillar labels and point labels may not agree. There is less disagreement between point semantic label and pillar semantic label in polar pillars so polar pillar segmentation miou has a higher upperbound (also reported in PolarNet paper).
For detection,the upperbounds are not 100 because some bboxes clutter around one pillar and one pillar can only represent one bbox so all other boxes are ignored or suppressed by NMS. In polar pillars this is more severe because polar pillar size far from the sensor is large and gets more boxes cluttered.
On the other hand, distortion discussed in Sec. 3.3 makes learning harder for detection using polar pillars.
- Range Stratified Convolution: this is extremely unclear for how the kernels are allocated for each grid.
Also relevant ablation studies are needed.
In Fig 2 we show an example of 3 stratums. The feature map has spatial size 128x128 on r-theta plane. The first dimension 128 is range( r) and second dimension is azimuth(theta). We divide range dimension into 8 stratums, each stratum of size 16x128. We apply convolution independently within each stratum.
We will amend the following ablation study to table 2.
| method | baseline | +range stratified conv | +range stratified conv&norm |
| -------- | -------- | -------- |-------- |
| det mAP    | 48.2    | 48.9     |49.1|
We also have the following ablation study for #stratum.
| #stratum      | 1 | 2 |4|8|16|
| ----------- | ----------- |----------- |----------- |----------- |----------- |
| det mAP     | 48.2 | 48.1| 48.8 | 49.1| 49.2|
- Section 5.2, multi-scale padding: It is still unclear why detection performance improves when the number of sectors increases. It might be because there are more overlapping bbox proposals, or NMS strategy. More detailed analysis is needed.
Is the reviewer asking why detection performance improves when the number of sectors increases for streaming generally or particularly for multi-scale padding?
For streaming, as discussed in Section 5.3, transformation of sectors to canonical coordinate system (coordinate system of the first sector) normalizes the data by reducing variability and thus makes learning easier.
We believe more overlapping box proposals does not help. We tried three NMS strategies:
1) local NMS: NMS within current sector and gather boxes for all sectors after NMS
2) stateful-NMS: gather boxes for current sector and previous sectors together before NMS and apply NMS to these boxes
3) global-NMS: gather boxes for all sectors together before NMS and apply NMS for all sectors
local NMS will result in more overlapping boxes because stateful-NMS and global-NMS suppress overlapping bboxes at the boundaries.
But local NMS is -0.5 mAP worse than stateful-NMS and global-NMS. The number of overlapping boxes does not matter. What matters is the quality of bboxes, i.e., whether the model has learned powerful features.
- directly accumulating 10 frames may incur localization error which may interfere with the detection result. Single frame baseline is also needed.
We accumulate 10 frames because using 10 frames is common practice on the nuScenes benchmark as in CenterPoint[35], HotSpotNet[6], CVCNet[5], PointPainting[29], PointPillars[15]. Single-frame results in very sparse point clouds and poor detection performance especially in velocity estimation (single frame det map 46.7 vs 50.6 for 10 frames). point clouds in previous frames are motion compensated, which reduces localization error.
- Not sure why the authors put emphasis on the point pillar backbone. Any other backbone can do the job.
We completely agree with your statement that any other backbone can do the job.
In fact, any other encoder (voxel based encoder vs pillar based encoder) will also work. Our main reason for choosing the PointPillars encoder and backbone was it's low latency and high performance which makes it very attractive for onboard applications.
We did not experiment with other encoders/backbones because it falls outside the scope of this work.
- For feature undistortion, why only apply this method on the classification head? The more appropriate way is to apply on the backbone. This experiment is needed.
The reviewer raised a good question. We did try feature undistortion in the backbone and below shows the comparison.
|  | det mAP | seg mIoU |latency|
| -------- | -------- | -------- |-------- |
| feature undistortion in backbone    | 49.9     | 70.6    | 55ms|
|feature undistortion in head|51.2|73.4|45ms|
Feature undistortion in backbone led to worse performance especially in semantic segmentation. The motivation for feature undistortion is trying to mimic cartesian representation because cartesian representation is better in detection (as discussed in previous reply).
But the backbone is shared by detection and segmentation heads and cartesian representation is worse in semantic segmentation.
So applying undistortion in backbone is not optimal.
Another reason we do not want feature undistortion in backbone is the feature maps are large in the backbone so adding feature undistortion will add a lot of computation and thus significantly higher latency.
- Section 5.3, diagnosis part: this part of analyzing two previous methods is not relevant to the main body.
While we greatly appreciate the reviewer's comments on the paper, we would like to respectfully disagree with the reviewer on this. We strongly believe that the analysis in Section 5.3 will be very useful for the research community as it gives them a complete perspective on how the streaming methods work, what are its advantages and what are its limitations. We believe that analysis sections like 5.3 broadens the readers' perspective which will help them devise even better solutions than ours in the future.
The paper tackles an important problem in the current literature around applications of ML to 3D computer vision, i.e. learning from 3D lidar data. The two methods that this work aims to improve are quite recent (2020) and to the best of my knowledge represent the state of the art  for sequentially processing portions of an entire lidar scan.
One limitation of this work is represented by the overall novelty of the method. Since the idea is to provide a representation of the input data that better fits the circular portions wrt. [11,13], this has an incremental nature wrt. the work in [11,13], since it can be regarded as an optimization of that idea.
Also the idea of undistorting feature to let a NN process a non-cartesian input representation is not novel but shares similarities with previous work [a,b]
The results provided in Table 1 show the usefulness of the proposed approach against [11,13], and the authors also provide a nice discussion trying to explain the reasons for this superiority in Section 5.3.
Nevertheless, extending Table 1 to include a comparison against the state of the art would have been useful to properly position the performance of the proposed approach with respect to the current literature. The results in Figure 5 go a bit in this direction and, from the provided charts, it seems like the current approach is still behind in terms of detection and semantic segmentation, while reports state of the art results in panoptic segmentation.
The paper is sometimes hard to read, due to the presence of typos as well as sentences that contains mistakes or are simply hard to understand. To improve the readability, especially around the methodology explanation, a careful review of the language used would be beneficial.
Taking everything into account I believe the paper falls short of requirements for acceptance especially with regards to the novelty and presentation aspects.
Additional points:
- line 118: point are accumulated from 10 successive frames: how did the author pick this specific parameter, and what is the weight of this value on the performance?
- line 181: the Range Stratified Normalization normalizes over individual regions within a certain range rather than on the entire spatial domain: how are these regions selected? It seems to me like the choice/number of regions should be triggered on the distance of objects and scene components with respect to the sensor. As these regions seem to be obtained by discretizing the spatial range, what happens to an object lying in between two regions? Would it receive two different normalizations?
References
[a] Yu-Chuan et al, Flat2sphere: Learning spherical convolution for fast features from 360 imagery. In: Proc. Conf. on Neural Information Processing Systems (NIPS) (2017)
[b] Coors et al, SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images, ECCV 2018.
Post rebuttal comment:
After going through the rebuttal and the comments from the other reviewers I feel my initial thoughts on the paper are confirmed. The method looks at an important problem but falls short in terms of novelty, clarity and experiments.
In particular I share the concerns with reviewer X4zp around the justification of polar coordinates stemming from the results presented in the paper, as well the readability issues that are raised across all reviewers. On a more minor note, the authors acknowledged in their rebuttal the lack of a more thorough review of related work around weight undistortion relatively to the citations that I suggested.
While I agree the method is substantially different from those in the literature, a proper reference to the state of the art in this particular field is important to bring the proposed contribution into context. I am thus confirming my initial rating.
No
We thank the reviewer for the review and insightful comments. We have addressed the points raised by the reviewer below:
- Novelty
Our major contribution is not polar representation. The first major contribution is to solve the limited spatial context issue for streaming.
Hence we proposed multi-scale context padding, which must be built on top of polar representation.
Secondly we dig into the limitation of polar representation for detection and address the challenges by range-stratified and feature undistortion. The distortion problem we addressed is similar to omnidirectional cameras in [a,b]. [a] adjusts sampling locations by heuristics and [b] adjusts kernel shapes at each row (also by heuristics). Their motivation is similar to ours, to undistort the features by some adaptive sampling strategy. It is normal to share the same motivation because we are trying to address similar issues but the approaches are substantially different. In feature undistortion we find the connection between convolution and bilinear sampling and automate the sampling process by convolution. Thanks for pointing to [a,b]. We were not familiar with 360 images. This shows that dealing with distorted data is a common issue and we will add them to related work for discussion.
- Clarity
Thank you for having a careful look and giving us your feedback. We will take another pass at the paper in the later version to improve the readability.
- point are accumulated from 10 successive frames: how did the author pick this specific parameter, and what is the weight of this value on the performance? the Range Stratified Normalization normalizes over individual regions within a certain range rather than on the entire spatial domain: how are these regions selected? It seems to me like the choice/number of regions should be triggered on the distance of objects and scene components with respect to the sensor. As these regions seem to be obtained by discretizing the spatial range, what happens to an object lying in between two regions? Would it receive two different normalizations?
using 10 successive frames is common practice for NuScenes as in CenterPoint[35], HotSpotNet[6], CVCNet[5], PointPainting[29], PointPillars[15], because single-frame results in very sparse point clouds and poor detection performance and high velocity error(single frame det map 46.7 vs 50.6 for 10 frames). In Fig 2 we show an example of 3 stratums. The feature map has spatial size 128x128 on r-theta plane. The first dimension 128 is range( r) and second dimension is azimuth(theta). We divide range dimension into 8 stratums, each stratum of size 16x128. Here is the ablation study of different #stratum following Table 2 and we can see a trend that adding #stratum helps detection. We choose 8 stratums for the balance of performance and #paramters in our model.
| #stratum      | 1 | 2 |4|8|16|
| ----------- | ----------- |----------- |----------- |----------- |----------- |
| det mAP     | 48.2 | 48.1| 48.8 | 49.1| 49.2|
If an object lies between two regions, it will receive two different normalization. This is desired because even within one object, the near end to sensor looks bigger and far end looks smaller. We need different normalization even within the same object.
Also we did not apply range-stratified convolution & normalization to the final layer, and the final regular 3x3 conv layer is able to leverage information from different normalization.
The paper is well-written and easy to follow. The proposed streaming LiDAR processing framework is technically sound and has achieved relatively good empirical results.
My primary concern of this paper is the justification for using polar coordinates. The authors have argued in the introduction (L27-L30) that using cartesian coordinates (and cuboid-shaped voxels) will waste both computation and memory.
However, this does not seem to be the case from the experimental results (Table 2 and Figure 1): the cartesian-based model has almost the same measured latency as the polar-based model.
At the same time, the polar-based model has inferior detection performance (1-2% in mAP) compared with the conventional cartesian-based model.
Therefore, the authors should explain their motivation behind switching to polar coordinates in their rebuttal.
The superior performance on the two segmentation tasks is not very surprising since similar observations have been made in Cylinder3D [Zhu et al., CVPR 2021].
Besides, the semantic segmentation results presented in this paper are more than 5% lower than the state-of-the-art submissions on the nuScenes leaderboard.
Thus, it is unclear whether their proposed solution is still effective when applied to these models with better performance.
Among the three downstream tasks, object detection is more important than the other two since it is usually deployed in the perception stack, while semantic/instance segmentation is usually used offline to build HD maps (where the latency does not matter as much). The authors should justify why they choose to perform these tasks jointly and whether this joint detection-segmentation task is a valid setting.
Finally, the authors should double-check their "et al." usages throughout the manuscript:
* L86, L94, and L269: missing ".".
* L236: misspelling as "el al".
There should also be a space between "et al." and the citation.
**Post-Rebuttal Comments:**
The authors' rebuttal has not addressed my major concern (i.e., the justification for using polar coordinates). As the use of a very powerful GPU is the main reason for the negligible speedup, it would be necessary to deploy these models on lightweight edge devices (such as Jetson AGX Xavier) to verify that the polar-based models are indeed faster than the cartesian-based models under a more realistic hardware setup. Without these results, it is unclear whether polar-based models are superior in the efficiency-accuracy tradeoff.
Therefore, I will keep my original score and vote for rejecting this paper.
References:
* Zhu et al., "Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation", CVPR 2021.
No
We thank the reviewer for their insightful comments and time spent reviewing the paper.
- justification for using polar coordinates
Cuboid-shaped voxels waste computation and memory because they use large feature maps than ours. Feature map size comparison is  shown below. For more than 8 sectors, cartesian pillars use twice the feature map size as ours because the way they partition the input region (figure 4 showed an example of 8 sectors, where for cartesian pillars half of input region is empty).
| #sectors    | 1 | 2| 4| 8| 16| 32|
| ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |
| Cartesian    | 512x512       |512x256|512x128|512x128|512x64|512x32|
|Polar  | 512x512       |512x256|512x128|512x64|512x32|512x16|
Here is the memory usage of the feature map ‘canvas’ as referred to in PointPillars. The memory is per sector in MB:
| #sectors | 1 | 2| 4| 8| 16| 32|
| ----------- | ----------- |----------- |----------- |----------- |----------- |----------- |
| Cartesian    | 33.6       |16.8|8.4|8.4|4.2|2.1|
|Polar  | 33.6      |16.8|8.4|4.2|2.1|1.3|
We are not seeing a noticeable improvement in runtime because we are measuring the runtime on a powerful V100 GPU, where feature map computation can run in parallel when gpu memory is not saturated.
However, installing V100 on an onboard pipeline may not be feasible due to its power consumption which would limit the battery range of the autonomous electric vehicle. The networks will need to be deployed on efficient embedded platforms like FPGAs where the increased feature map size of cartesian sectors will result in increased memory and latency.
On the other hand, polar representation enables multi-scale context padding, an effective and efficient fix for the major limitation of streaming - limited spatial context of each sector.  Previous streaming papers both focus on solving this issue but neither worked as well as context padding.
- it is unclear whether their proposed solution is still effective when applied to these models with better performance
We also tried our full-sweep polarstream with the same 3D ResNet-like encoder as in CenterPoint (named PLS1 heavy in the following table) and the following table is what we got for detection and semantic segmentation on NuScene validation set. Ours-PLS1 heavy outperformed Cylinder3D with a lower latency on NuScene validation set. In our work we focus on onboard applications like streaming and those heavy backbones cannot run onboard so we choose the encoder and backbone from the PointPillars [15] model.
| Methods | det mAP | seg mIoU | runtime | #parameters|
| -------- | -------- | -------- | -------- | -------- |
| CenterPoint    | 56.4     |      | 11Hz   | 149MB|
| Cylinder3D | | 76.1 |11Hz(reported) 2Hz(reproduced)| 215MB|
| Ours-PLS1 | 51.2| |26Hz|65MB|
|Ours-PLS1 | |73.8|34Hz|65MB|
|Ours-PLS1 heavy|56.2| |11Hz|149MB|
|Ours-PLS1 heavy||76.8| 15Hz|149MB|
- justify why they choose to perform these tasks jointly and whether this joint detection-segmentation task is a valid setting.
Autonomous driving is still a largely unsolved problem and it is definitely an open research question on whether online perception via detection or semantic segmentation tasks is more favorable to the downstream tracking and planning modules of an AV.
Other than detecting 3D boxes directly, an equally plausible perception pipeline is doing lidar segmentation, differentiating foreground lidar points against drivable surface, clustering those lidar points and tracking those clusters. The above-mentioned pipeline might also be better suited for detecting irregularly shaped generic objects like tree branches fallen on the road where a 3D box might not be the best representation for them.
Our work also shows that detection accuracy improves when jointly trained with semantic segmentation. A combination of both can produce more reliable perception results for downstream tasks.
- typos
Thank you for having a careful look and finding out these typos. We will make sure to correct them in the revision. We will also have a careful look at the paper again to make sure that any more such typos don't exist.

Meta-review:
role: PC
data: comment: This paper proposes a method for object detection and segmentation in LIDAR data. The reviewers generally appreciated the importance of the problem and the efforts made in the paper. However, several issues where raised on novelty, writing and experiments.

The authors' response could not address the most critical concern, which was the justification of polar coordinates. The reviewers judged that this claim was not backed up by the experiments, and a consensus for rejection emerged.

I concur.

Decision: Accept (Poster)