Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper studies streaming algorithms for low-latency LiDAR object detection and segmentation. The authors propose to use the polar coordinate system to represent the point cloud data in order to reduce memory and compute waste. They further introduce two techniques to bridge the performance gap between polar and cartesian-based solutions: (1) multi-scale padding from neighboring sectors to increase the spatial context, and (2) feature undistortion and range stratified convolutions to address the distortion. The authors have evaluated their proposed model on the nuScenes dataset and have achieved good empirical performance.
main_review: The paper is well-written and easy to follow. The proposed streaming LiDAR processing framework is technically sound and has achieved relatively good empirical results.

My primary concern of this paper is the justification for using polar coordinates. The authors have argued in the introduction (L27-L30) that using cartesian coordinates (and cuboid-shaped voxels) will waste both computation and memory. However, this does not seem to be the case from the experimental results (Table 2 and Figure 1): the cartesian-based model has almost the same measured latency as the polar-based model. At the same time, the polar-based model has inferior detection performance (1-2% in mAP) compared with the conventional cartesian-based model. Therefore, the authors should explain their motivation behind switching to polar coordinates in their rebuttal.

The superior performance on the two segmentation tasks is not very surprising since similar observations have been made in Cylinder3D [Zhu et al., CVPR 2021]. Besides, the semantic segmentation results presented in this paper are more than 5% lower than the state-of-the-art submissions on the nuScenes leaderboard. Thus, it is unclear whether their proposed solution is still effective when applied to these models with better performance.

Among the three downstream tasks, object detection is more important than the other two since it is usually deployed in the perception stack, while semantic/instance segmentation is usually used offline to build HD maps (where the latency does not matter as much). The authors should justify why they choose to perform these tasks jointly and whether this joint detection-segmentation task is a valid setting.

Finally, the authors should double-check their "et al." usages throughout the manuscript:
* L86, L94, and L269: missing ".".
* L236: misspelling as "el al".

There should also be a space between "et al." and the citation.

**Post-Rebuttal Comments:**
The authors' rebuttal has not addressed my major concern (i.e., the justification for using polar coordinates). As the use of a very powerful GPU is the main reason for the negligible speedup, it would be necessary to deploy these models on lightweight edge devices (such as Jetson AGX Xavier) to verify that the polar-based models are indeed faster than the cartesian-based models under a more realistic hardware setup. Without these results, it is unclear whether polar-based models are superior in the efficiency-accuracy tradeoff. Therefore, I will keep my original score and vote for rejecting this paper.

References:
* Zhu et al., "Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation", CVPR 2021.
limitations_and_societal_impact: This paper does not have noticeable negative societal impacts. As for limitations, their proposed solution does not seem to work well for LiDAR detection, which is arguably much more important than the two segmentation tasks.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: The paper proposes some modifications to the work of [11,13], which processes 3D lidar data by analysing consecutive sectors of a 360 deg. scan as soon as they are acquired. In particular, it suggests using polar coordinates rather than cartesian as a way to more efficiently adapt to the sectors, which are circular portions. This reduces the memory required to process the lidar data within a neural network, but requires specific additional stages in the neural network itself to take care of the different shape and coordinate system of the input data. The authors hence propose specific stages to take care of this, namely feature undistortion and range stratified convolution and normalization. Results are ablated and compared against [11,13] + a cartesian-based equivalent of their method on the NuScenes benchmark.  
main_review: The paper tackles an important problem in the current literature around applications of ML to 3D computer vision, i.e. learning from 3D lidar data. The two methods that this work aims to improve are quite recent (2020) and to the best of my knowledge represent the state of the art  for sequentially processing portions of an entire lidar scan. 

One limitation of this work is represented by the overall novelty of the method. Since the idea is to provide a representation of the input data that better fits the circular portions wrt. [11,13], this has an incremental nature wrt. the work in [11,13], since it can be regarded as an optimization of that idea. Also the idea of undistorting feature to let a NN process a non-cartesian input representation is not novel but shares similarities with previous work [a,b]

The results provided in Table 1 show the usefulness of the proposed approach against [11,13], and the authors also provide a nice discussion trying to explain the reasons for this superiority in Section 5.3. Nevertheless, extending Table 1 to include a comparison against the state of the art would have been useful to properly position the performance of the proposed approach with respect to the current literature. The results in Figure 5 go a bit in this direction and, from the provided charts, it seems like the current approach is still behind in terms of detection and semantic segmentation, while reports state of the art results in panoptic segmentation. 

The paper is sometimes hard to read, due to the presence of typos as well as sentences that contains mistakes or are simply hard to understand. To improve the readability, especially around the methodology explanation, a careful review of the language used would be beneficial. 

Taking everything into account I believe the paper falls short of requirements for acceptance especially with regards to the novelty and presentation aspects. 

Additional points:
- line 118: point are accumulated from 10 successive frames: how did the author pick this specific parameter, and what is the weight of this value on the performance?  
- line 181: the Range Stratified Normalization normalizes over individual regions within a certain range rather than on the entire spatial domain: how are these regions selected? It seems to me like the choice/number of regions should be triggered on the distance of objects and scene components with respect to the sensor. As these regions seem to be obtained by discretizing the spatial range, what happens to an object lying in between two regions? Would it receive two different normalizations?

References
[a] Yu-Chuan et al, Flat2sphere: Learning spherical convolution for fast features from 360 imagery. In: Proc. Conf. on Neural Information Processing Systems (NIPS) (2017)
[b] Coors et al, SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images, ECCV 2018.


---
Post rebuttal comment:

After going through the rebuttal and the comments from the other reviewers I feel my initial thoughts on the paper are confirmed. The method looks at an important problem but falls short in terms of novelty, clarity and experiments. In particular I share the concerns with reviewer X4zp around the justification of polar coordinates stemming from the results presented in the paper, as well the readability issues that are raised across all reviewers. On a more minor note, the authors acknowledged in their rebuttal the lack of a more thorough review of related work around weight undistortion relatively to the citations that I suggested. While I agree the method is substantially different from those in the literature, a proper reference to the state of the art in this particular field is important to bring the proposed contribution into context. I am thus confirming my initial rating.
limitations_and_societal_impact: I have no specific concerns in these regards.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: In this manuscript, the authors propose streaming based LiDAR detection using polar grid representation. Incorporating dedicated convolution and undistortion methods, the proposed method shows good performance on the NuScenes dataset.
main_review: The main issue of this manuscript is lack of clarity and analysis which makes the claimed contribution unclear:
* In figure 1, the authors claim polar representation is better than cartesian in terms of memory and computation. However, no such comparison is provided
* There is overall no result visualization or case analyses (segmentation+detection) to demonstrate the usefulness of the proposed method.
* For table 1, the detection result for <=4 sectors are worse than Cartesian, even with CP. Why is this the case? This seems to contradict with the claim that Polar representation is better. And why segmentation tasks do not have such an effect? Insights are needed.
* Range Stratified Convolution: this is extremely unclear for how the kernels are allocated for each grid. Also relevant ablation studies are needed.
* Section 5.2, multi-scale padding: It is still unclear why detection performance improves when the number of sectors increases. It might be because there are more overlapping bbox proposals, or NMS strategy. More detailed analysis is needed.

In addition, only running the experiment on the NuScenes dataset is not sufficient in my opinion. I suggest the authors add the Waymo dataset. Also directly accumulating 10 frames may incur localization error which may interfere with the detection result. Single frame baseline is also needed.

Some other issues:
* Not sure why the authors put emphasis on the point pillar backbone. Any other backbone can do the job.
* For feature undistortion, why only apply this method on the classification head? The more appropriate way is to apply on the backbone. This experiment is needed.
* Section 5.3, diagnosis part: this part of analyzing two previous methods is not relevant to the main body. 

Evaluation
Originality: the method is original
Quality: the proposed method and experiments are technically sound.
Clarity: main issue is the clarity, mentioned in my comments
Significance: the low latency LiDAR detection model is of interest to broad audience in the community.


================post rebuttal====================

I would like to thank the authors for the detailed reply. While the reply addressed some of my concerns, I am still not convinced by 1) the memory and computation advantage, and 2) the performance gain vs. the Cartesian coordinate (e.g., only larger sectors work, but sacrifices the overall performance). Also the explanation of the solved cases and Range Stratified Convolution is still vague. My score remains the same.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper proposes a method for object detection and segmentation in LIDAR data. The reviewers generally appreciated the importance of the problem and the efforts made in the paper. However, several issues where raised on novelty, writing and experiments.

The authors' response could not address the most critical concern, which was the justification of polar coordinates. The reviewers judged that this claim was not backed up by the experiments, and a consensus for rejection emerged.

I concur.
Decision: Accept (Poster)