PolarStream: Streaming Object Detection and Segmentation with Polar Pillars
Authors: Qi Chen
Sourabh Vora
Oscar Beijbom
Venue: NeurIPS 2021 Poster (2021)
Keywords: Streaming Object Detection
3D Object Detection
LiDAR Segmentation
3D Panoptic Segmentation
Polar Grid

Abstract: Recent works recognized lidars as an inherently streaming data source and showed that the end-to-end latency of lidar perception models can be reduced significantly by operating on wedge-shaped point cloud sectors rather then the full point cloud. However, due to use of cartesian coordinate systems these methods  represent the sectors as rectangular regions, wasting memory and compute. In this work we propose using a polar coordinate system and make two key improvements on this design. First, we increase the spatial context by using multi-scale padding from neighboring sectors: preceding sector from the current scan and/or the following sector from the past scan. Second, we improve the core polar convolutional architecture by introducing feature undistortion and range stratified convolutions. Experimental results on the nuScenes dataset show significant improvements over other streaming based methods. We also achieve comparable results to existing non-streaming methods but with lower latencies.

PolarStream: Streaming Lidar Object Detection and Segmentation with Polar Pillars
Qi Chen'
Sourabh Vora Motional Santa Monica, CA 90401
Johns Hopkins University Baltimore, MD 21218 gchen420jhu. edu
sourabh. vora@motional. com
Oscar Beijbom Motional
Santa Monica, CA 90401
oscar bei jooo@motiions com
Abstract
Recent works recognized lidars inherently streaming data source and showed that the end-to-end latency of lidar perception models can be reduced significantly by operating
sedge-shaped point cloud sectors rather then the full point cloud. However, due to use of cartesian coordinate systems these methods represent the sectors rectangular regions, wasting memory and compute. In this work we propose using polar coordinate system and make two key improvements on this design. First, we increase the spatial context by using multi-scale padding from neighboring sectors: preceding sector from the current scan and/or the following sector from the past scan. Second, improve the core polar convolutional architecture by introducing feature undistortion and range stratified convolutions. Experimental results on the nuScenes dataset show significant improvements over other streaming based methods. We also achieve comparable results to existing non-streaming methods but with lower latencies.
Introduction
The ability to accurately perceive objects in dense urban environments still remains challenging problem for self-driving cars. While such self-driving cars typically deploy wide variety of sensors lidars play a key role due to the accurate range information provided. Driven in part by the availability
271. the last decade has seen tremendous progress in lidar based 3D 10]. However, these methods all ignore the fact that most lidar sensors scan the scene sequentially as the lidar rotates around the 7-axis. They instead wait for the rotational scan to complete (colloquially known as full sweep) before processing data, thereby introducing large data capture latency (usually 50 100 ms).
of benchmark datasets 22
3
object detection 39
16
32
21
First, Han et al. 133 and then STROBE [1 recognized this problem and proposed solutions which processed lidar sectors (shown in Fig. M soon as they arrived. They showed that streaming based architecture can achieve significantly reduced latency over the traditional non-streaming baselines. Both of these methods encode the point clouds as an image in bird's s-eye view (BEV) using cuboid- shaped voxels. In doing so, they ignore the natural polar representation formed by the lidar sectors. Using cuboid-shaper voxels restricts them to performing convolutions on the minimal rectangular region enclosing the point cloud sector which wastes both computation and memory. As shown in Fig.
large portion of the enclosed rectangular region remains empty.
"work done while interning at Motional
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

700
Previous work
67.5
Panoptic Quality
65.0
62.5
57.3
Han et STROSE
Ours
52.
PolorStream
30 60
90
End-to-End Latency / ms
Figure 1: Left: An illustration of streaming lidar point clouds bird's eye view. Lidar point clouds arrive as wedge-shape sectors (shown in gray masks) as the scanner rotates. Previous methods, Han al. [33 and STROBE(L], represent the sectors using rectangular regions, wasting half of memory and computation for empty regions. Ours represents the sectors as wedge-shape regions using a polar grid. Right: Comparison of different streaming methods wrt. Panoptic Quality vs End-to-En End Lantency as we slice the full sweep into n = 2, 4, 8, 16, 32 sectors using the NuScenes val split. The end-to-end latency includes 50/n ms for LiDAR scan and the total runtime of the algorithms.
Another challenge associated with streaming perception models the limited view of the scene observed by each sector. Objects close to the ego- ago-vehicle can often be fragmented across multiple sectors as shown by the car highlighted in green in Fig Han et al. [3] proposes to increase the context available to the model by maintaining recurrent memory across consecutive sectors. STROBE [ also aggregates representations from the previous sectors by maintaining full-sweep feature maps across multiple scales. However, both these solutions add extra computation.
In this work, we propose to encode individual point cloud sectors using polar pillars. Polar pillars naturally address the inefficiency of existing streaming approaches by representing the point cloud sectors as more compact wedge-shaped regions as shown in Fig. []] Further, we propose a simple Minimall-latenee approach to enhance the context available to the model by simply padding the representation of the neighboring sectors across multiple strides of the backbone. Using polar pillars allows us to pad features from the preceding sector of the current scan and/or the following sector from the previous scan, no matter how many sectors the full sweep divided into.
The polar BEV representation has recently started gaining attention in the lidar perception literature primarily because it balances the points across grid cells. In fact, polar grid outperforms the cartesian grid on the lidar segmentation task [36 37.. However, the detection peformance
polar grid still lags the cartesian grid [ 5 (23]. This is because of the distortion the objects undergo when this representation is ultimately unfolded to rectangular representation to enable the use of convolutional layers. The object represented by the green box in Fig. 2
shows example of this distortion. Further, the distortion increases with range as the pillars progressively become larger. This makes a polar representation compatible with the ianislatioonivvriiann property of convolution.
In this work, we propose several techniques to address the distortion problem described above. We first propose Feature Undistortion module which transforms the polar representation into canonical Cartesian representation (as shown in Fig. (2 for classification branch. Next, we propose using the Range Stratified detection head. These layers apply different convolution kernels and normalization based on range (Fig to cater the changing pillar sizes in a polar grid. Our proposed model closes the gap on 3D object detection models using cartesian representations without adding any significant latency.
layers on the regression branches of the
Finally, train multitasking streaming models that do simultaneous 3D object detection. lidar segmentation and panoptic segmentation, for the first time in literature. Results on the nuScenes dataset show that our proposed model PolarStream outperforms all streaming methods in both panoptic quality and speed. PolarStream also stays competitive with the top-performing lidar perception methods the nuScenes leaderboard while being least twice fast as the rest. We do several ablation studies and extensive analysis to show the effectiveness of PolarStream.
In summary, contributions are:

---

An efficient streaming based lidar perception models using polar grid.
Multi-scale context padding: an efficient approach to enhance the context of streaming lidar perception models
Several improvements to the core problem of applying convolutions on a polar grid: Feature Undistortion, Range Stratified Convolution&Normalizattt all add minimal lateney to our model.
2 Related Works
2.1 Non-streaming lidar perception
Most lidar perception architectures take inspiration from the image perception literature 224
18 16 32
17
Some single-stage methods typically convert the point cloud into bird bird's-eye view image 33
or range view image (21 [O] and perform detection in those views. The most common paradigm is to convert the lidar point cloud into a BEV image as offers several advantages like a lack of scale ambiguity, near lack of occlusion, the ease of fusing HD maps [31] and performing simultaneous
detection and trajectory predictions [4] [9]. To convert the point clouds into a BEV representation, most existing models choose group the points into voxels. The most commonly used voxels
are cuboid-shaped based on Cartesian coordinates. Vox (20). MV3D (7]. Pixor (33, Complex- YOLO (26) represent the saboid-shaped voxels as occupancy grids. To avoid quantization effects of occupancy grids and extract richer voxel features, VoxeINet [39] samples a fixed number of points within each voxel and applies simple PointNet [22] them. For efficiency, PointPillars [T6] discretizes the 3D space into pillars so there is only one voxel along the height dimension.
Some recent methods that operate on BEV start to explore polar voxels for point clouds. For 3D) object detection, Alsfasser et al []] voxelizes points under the Cylindrical Coordinate System, MVF 38 adopts both cuboid-shaped voxels and spherical voxels, and CVCNet combines cylindrical and spherical coordinate system into one Hlybrid-Cylindriial-SSheee
(HCS) coordinate system to detect object from both bird's 's eye view and range view. On the other hand, the success of PolarNet [36] and Cylinder3D (37] shows the advantage of Cylindrical grids over Cartesian voxels in LiDAR semantic segmentation. Panoptic-Polate PolarNet 40) further extends PolarNet to the task of panoptic segmentation.
2.2 Streaming lidar perception
Streaming lidar perception is relatively new in literature and offers compelling argument in reducing the end-to-end latency. Han et al [13] proposed a couple of enhancements to convert a 3D object detector to operate streaming data: a) using an LSTM to accumulate features from preceding sectors and b) applying stateful NMS to suppress objects across multiple sectors. STROBE [T] accumulates features not only from the preceding sectors of the same scan but also from the previous scan by maintaining multi-seale memory feature maps. Features extracted from the current sector is concatenated and fused with the corresponding cropped region in the memory feature maps.
PolarStream
In this section, we introduce PolarStream, streaming model based on polar pillars. We introduce how we prepare lidar streaming data in Sec3.1] polar pillars as representation for point clouds sectors in See 3.2 the simultaneous detection and segmentation model including techniques to improve detection on a polar grid in 3.3 3.3 and multi-scale context padding to enlarge context in Sec 3.4 3.1 Streaming LiDAR Inputs
Since there is no streaming lidar dataset available, we simulate a streaming system from the NuScenes dataset [3] by slicing the point clouds into sectors according to their azimuth. As shown in Fig [ each sector is like a slice of a full pizza. We try n = 1, 1,2, 4, 8, 16, 32 sectors in our experiments, where = means full sweep. The dataset contains 1, 000 scenes, comprising 700 scenes for training, 150 scenes for validation and 150 scenes for test. Each scene is of 20s duration, captured by 32-beam lidar. 40, 000 frames are annotated in total, including 10 object categories such cars, motorcycles and pedestrians and six stuff classes such as vegetation and drivable region. We consider 10 object classes for detection, 16 classes in total for semantic and panoptic segmentation.
3.2 Polar Pillars
The point clouds sector consists of N points, each represented by a vector of point feature f, (rp Op Zp, zp:Tps Tp Ups ip. (p), where (xp Up, zp) is its Cartesian coordinates. (rp, is the polar coordinates. i, is the reflection intensity and lp is the timestamp when the lidar point is captured. Points are accumulated from 10 successive frames in total to obtain denser point clouds. The points from previous frames are compenstem and transformed to current frame. We group the
3

---

Clouds Poler Pilaars
polar grid points
Classification Head
Interpolated cartesian grid points
Pillar Feature Encoder
Backbone (2D CNN)
3D Box
Regression
3D
Heads
Feature Undistortion
Unfold in BEV
Upsample
Concat
separate kernels
R
Segmentation
Head
50
-202
Range Straiified Comvoluionn&Nommllzaiioo
Figure 2: Simultaneous LiDAR object detection and segmentation network with polar pillars. We adopt the same backbone as in PointPillars[16 [16], and add a semantic segmentation head in parallel with the detection heads. The input wedge-shape pillars are unfolded into rectangular feature map for convolution. The object (green box) is distorted because one end near the sensor looks bigger and the other end far from the sensor looks smaller. Feature Undistortion is applied to classification head to mimic bilinear sampling and interpolate cartesian pillar features from polar pillar features. Range Stratified Convolution& Normalization is applied to center offset regression head.
points according to the cylindrical pillar resolution (òr, 80, 6z) where d2 Zmax Zmain SO there is only one pillar along the height dimension. Following MVF [8). we adopt dynamic voxelization to sample all points within each pillar, instead of randomly sampling a fixed number of points per pillar.
3.3 Simultaneous Detection and Segmentation
We design PolarStream: a simultaneous object detection and segmentation network by extending PointPillars [66], one of the most widely used 3D object detectors balancing accuracy and speed. As shown in Fig 2 PolarStream consists of Pillar Feature Encoder, followed by a 2D CNN backbone
and U-Net 25 like structure. On top are the detection and segmentation heads.
Detection Heads We adopt CenterPoint (35 heads with modifications make it compatible with polar pillars. To assign targets to the 10-class heatmap to indicate the objects, the gaussian radius of the object center is computed using the span of range and azimuth of the object bounding box, instead of using length and width of the box. Following CenterPoint, we also regress the center offset as da, dly the bounding box size 1, w., h as log 1, log w. log h. and predict the bounding box height z. We regress the relative bounding box orientation Φ as cos σ, sin ø and relative velocity as uzs Uy similar to (23). Unlike most methods, which use multi-group detection heads that partition object classes to several groups according to their size, we use singlc-grous group detection heads to balance accuracy and speed. A comparison against multi-group detection heads shown in Supplementary. For streaming data with n > 1, we apply stateful-NMS proposed in Han et al. [3]
Segmentation Head To extend PointPillars for segmentation, head in parallel with the detection heads. The segmentation head is made of a single 1x convolution layer. The input for the segmentation head is concatenation of the outputs from pillar feature encoder and bilinearly upsampled features from the 2D backbone.
add semantic segmentation
Panoptic Fusion Similar to Panoptic- Polaanet [40], for each point belonging to things, we predict the instance id as the box id whose category is the same and center is the nearest. For streaming data with > 1, the panoptic segmentation task is not well defined. For example, the points in the det sector may belong to the box in the (i + 1),A sector the majority of the box is in the (i + 1) sector. However, when are doing panoptic fusion for its. sector, we do not have information from the (i + 1), sector. Therefore we choose global panoptic fusion for streaming point clouds, i.c., we assign instance ids according to the boxes from all sectors of the same sweep.
Multi- Learning We adopt Focal Loss [1 for classification and LI loss for bounding box regression, orientation and velocity estimation. For segmentation, use the weighted cross-entre cross-entropy loss and lovasz-softmax loss 2] The total loss is the weighted sum of losses for each component.
Feature Undistortion As mentioned in See objects have distorted appearances with polar pillars, we propose Feature Undistortion to undistort the features. As shown on the top right of Fig the idea of undistortion is to interpolate features at cartesian pillar locations from the original polar pillar

---

locations so that the translation translaton-invariint property of convolution applies. We find the connection of bilinear sampling to convolution and mimic bilinear sampling using convolution. For bilinear sampling, the interpolated features at point p can be sampled from its neighboring points.Vp
Jp
Wk Ph
PEENP
where wi is a function of distance(p. pk).
We find Equutioon]aas the similar form to convolution, except that for convolution wi is fixed because same kernel is slided through every location of the feature map. To make wy distance-dependent, we tweak Equation 1 by adding a new parameter wi so Equation can be rewriten as:
Sp
www.sSr
(2)
PEENP
where w/, is conditioned on distance(p, P&). We model w/ as the output of a neural network. We build standalone fully convolutional network g that takes position encodings at ps and its neighboring points Np i.e {pe, = (ra, cos 0, sin 0,, Zi, D; E Ny.. U Up J as input, and output wy.- Simply put:
w/ee = s((ee)))
To make it more general, we also add a bias term 1/g and another standalone network so that b/2 a( (pee]) and
fp Σ
wu(Wffn 6k)
PLEN
I and 9 is trained together with our main network, and during inference w/. and b/g are fixed for each location so it does not need extra runtime for and q. We apply feature undistortion in center heatmap prediction.
Range Stratified Convolution & Normalization Another challenge with polar pillars that the center offset is dependent on range and azimuth so it has different statistics at different regions: suppose the heatmap center is at (re, fz), and the target is at (rt. 0e). The center offset is
d₂ = Tı cos 0 - Te cos 0g dy = r; sin A/ re sin 6.
(5)
For simplicity, assume 7t- = re, ie. the center offset moves along a circle. Suppose 02 > Ag then 8 = 0g + 0, < 6g 80
(6)
where 0 is small angle and 80 is the polar pillar angle size. Then
cos 0.) & -re0s Og d₂ (cos - 0.) coo
range and azimuth and observe that for Cartesian pillars center offset ranges from 1 to 1 and mean is 0.49 and std 0.28, while polar pillars center offset ranges from -2 to 2, mean is 0 and std is 0.64. The polar std is much larger than that for Cartesian pillars. Hence it's more difficult to regress center offset based on polar pillars. Based on these observations, convolution and batch normalization 13). As shown on bottom right of Fig Range Stratified Convolution applies individual kernels different ranges and Range Stratified Normalization only normalizes over individual regions within certain range instead of entire spatial dimension. We apply Range Stratified Convolution&Normaliii Stratified Normalization to the shared convolution for detection heads.
Similarly, we can derive that d, is also dependent
propose Range Stratified Convolution& Normalization instead of regular
to center offset regression. We also apply Range
3.4 Multi-Scale Context Padding
Trailing- Edge Context Padding As shown in Fig the sector unfolded to a rectangle feature map on r-0 plane as input for convolution. The lidar sectors arrive one after another by increasing the angle the sensor scans so the unfolded feature map of sector is spatially connected to its preceding sector along 0 dimension. This unique property of using polar pillars inspires to, instead of zero- padding along 0 dimension, pad the features from preceding sector where is spatially connected to current sector. The receptive field of neuron increases as the neural network goes from bottom layer to top and the network encodes multi-scale representation of the input at different stages. This motivates us to pad context from preceding sector before every convolution of the 2D CNN backbone,
as illustrated in trailing-edge padding of Fig Although we only pad a few columns to the feature map, the neural network is replenished with sufficient context from multiple ranges and multiple scales at different stages of the network. We keep aero-padding for r dimension and the other end of the 0 dimension, as the other end of 0 dimension points to the future sector.
5

---

Trailing-adge padding
Backbone (2D CNN)
1=2/8
1=0
t- 1/6
R
ped
ped
Bidirectional padding
merge
marge
1=0
t=1
warp
0
conv
t-7/6
R
deconv
pad
ped
Figure 3: Multi-Scale Context Padding. We present both trailing-edge padding and bidirectional padding. Trailing padding pads current sector with features from preceding sector. Bidirectional padding additionally pads current sector with features from 'following' sector of past time frame. Full-sweep feature maps merged for past time frame and warped the coordinate system of current time by ego-motion compensation. Context Padding is applied to every convolution in the backbone.
Bidirectional Context Padding With trailing-edge padding the current sector is padded with context from preceding sector. To provide further context we pad the leading-edge with warped features from the following sector of the previous sweep. To do this aggregate the full-sweep multi-scale feature maps from the previous sweep and warp the feature maps to the coordinate system of current sweep using ego-motion compensation. We then pad the leading edge of the current sector with the corresponding warped features spatially connected to the current sector.
4
Implementation
4.1 Network Details
For polar pillars with n sectors per sweep, r, 2, 2 range is 30.3),m [-3.1488, -3.1488
6.2976/n] rad and [-5,3]m 3,, the pillar size is (0.098, 0.0123, 8). For Cartesian pillars, the pil- lar size (0.2, 0.2, 8). When n = 1, x, y, range leaving same input size of 512 512 1 for both Cartesian pillars and polar pillars when 1. We find the minimal rectangular region to enclose the sectors when n > for Cartesian Pillars. We set segmentation loss weight to 2 and classification loss for both polar pillars and Cartesian pillars. For Cartesian pillars the bounding box regression weight is 0.25. For polar pillars, since regression is harder, we the loss weight to 0.5. We make sure they the best configuration for each setting. For and in Feature Undistortion, they share the same architecture: a 3x3 conv followed by 1x1 conv with tanh as activation. We show the network architecture in Supplementary. All runtimes are measured on single V 100 GPU using Pytorch.
51.2, 51 .2mm, [-51.2,51 21 m and [-5, m
4.2 Augmentation
We adopt class- balanced sampling proposed in CBGS [1 sectors, we conduct random flipping along x, y axes, scaling with a scale factor sampled from [0.95, 1.05], rotation around z axis between [-0.3925, 0.3925] rad and translation in range [0.2, 0.2, 0.2) m x, y, 2 axis. Unlike most methods, we do not use database sampling 30 for fast training.
Before slicing the point clouds into
Experiments and Results
5.1 Evaluation
We gather the predictions from individual sectors and evaluate PolarStream similar to full-sweep meth- ods. We evaluate 3D detection and lidar semantic segmentation on the NuScenes benchmark [3. The detection mean average precision (mAP) based on the distance threshold (i.e. 0.5m, 1.0m, 2.0m and 4.0m). Additionally, nuScenes detection score (NDS) (3), weighted sum of mAP and

---

Table 1: Comparison of streaming methods on nuScenes Val split. CP: context padding; CP xl:
trailing-edge padding; CP x2: bidirectional padding.
Panoptic Quality (PQ)
Segmentation mloU
Method
4
16
2
16
Cartesian Pillars
67.8
66.8
67.6
65.5
64.1
59.8
72.1
71.5
70.9
70.2
68.6
65.7
Han al.
66.3
67.4
66.2
64.6
61.6
70.6
71.5
70.1
69.2
66.5
STROBE
63.8
65.2
66.2
65.2
62.9
59.5
69.2
69.7
69.8
69.6
67.6
65.6
Polar Pillars
68.7
68.6
67.7
66.2
63.9
60.3
73.4
73.4
72.5
70.8
69.5
67.1
Ours- PolaarSSttt
68.8
69.6
68.8
69.2
68.3
73.5
74.2
73.4
73.8
73.1
Detection mAP
Detection NDS
Method
4
16
2
16
Cartesian Pillars
52.3
51.3
54.9
52.4
47.6
60.7
60.3
62
57.9
59.5
57.1
Han al. 13
50.9
52.9
53.8
52.7
50.6
59.6
60.3
60.8
60.3
58
STROBE
46.9
48.7
49.4
47.9
45.4
42
53.8
54.6
51.5
48.9
47.1
44.8
Polar Pillars
51.2
52.1
51.9
52.5
51.9
46.7
60
61.4
61.4
60.9
60.3
55.8
Ours-Polle PolarSSoarSt
51.5
53.2
52.7
53.9
52.4
60.3
61.2
60.9
61.4
60
precision on box location, scale, orientation, velocity and attributes. For semantic segmentation, we follow the standard mean intersection-ovve--uiioo (mloU) metric. Since nuScenes does not provide instance labels for panoptic segmentation, we follow Panoptic-PolarNer [40] to generate labels and evaluate panoptic segmentation on validation split using the Panoptic Quality (PQ) metric.
5.2 Comparison with other Streaming Methods
Baselines Han et al. and STROBE([]] did not release their code and in addition performed evaluation on two different datasets. To enable benchmarking we re-implemented their methods using the same backbone and input resolution we re-implemented stateful-NMS and stateful-RNN STROBE. We did not implement the HDD map branch in STROBE in order ensure fair comparison. We also apply stateful-NMS and global panoptic fusion to all the methods in comparison as they just post-processing techniques. We extend both methods the task of simultaneous object detection, semantic segmentation and panoptic segmentation. We also provide baselines that simply apply Cartesian pillars or polar pillars individual point clouds sectors. We compare panoptic quality, segmentation mIoU, detection mAP, NDS with the baselines using n = 1, 2, 4, 8, 16, 32 sectors [T. We also show the comparison of our method Han et al. and STROBE wrt. PQ vs. end-to-end latency (Fig. I.
use and evaluated the nuScenes dataset. Specifically
Han el al. and multi-scale memory module in
Results Tab. shows that PolarStream outperforms all previous streaming methods, including the Cartesian pillars baseline, in both PQ and Segmentation mloU. When there are more than four sectors in scene, previous methods as well as the Cartsian/polar pillar baselines show trend of decreasing PQ and mloU as the number sectors increases. However, our PolarStream with bidirectional context padding does not show such trend: the performance remains almost the same or even better than the full-sweep method. When n = 1, PolarStream got +0.9 and +1.3 improvement in PQ and segmentation mloU compared to the Cartesian pillars baseline. When sectors become smaller and spatial context becomes limited, the improvement is more significant. When n = 32, our PolarStream with Bidirectional Context Padding outperforms all previous streaming methods by large margin, with +6.7 and +6.6 improvements in PQ and segmentation mloU. This shows that our bidiretional context padding makes better use of spatial context compared to previous methods. Our detection NDS is always the highest or at least on par with the highest among all streaming models. Interestingly, Cartesian pillars/Han et al.'s method show higher mAP than ours for 1, 4 and sectors, when the sectors have plenty of spatial view and our context padding does not have many benefits. Our PolarStream outperform all previous streaming methods detection mAP for and 32 sectors, when spatial view is limited and Bidirectional Context Padding shows more advantages. In addition, the orientation and velocity error of Han et 's is on average 14.6% and 12.9% higher than ours, which will cause problems for the downstream tracking and prediction tasks. As shown in Fig our
7

---

pooied featuros from
festures from
n
B
Concat
Input
Teatures from current soctor
notis bdded
podded featume from precesling
paskked TealLTES 56.30
Figure An illustration of how previous streaming methods, Han et al. 13 and STROBE II| enlarge spatial context of current sector.
methods offers better operating points considering both accuracy and end-to-end latency. Detailed metrics including velocity error and per-class metries are shown in Supplementary.
5.3 Discussions on Streaming
Full Sweep vs. Streaming Contrary to the findings of Han et al [3], we saw improved detection models trained on full-sweeps. We
performance using 2, 4, 8 and 16 sectors as compared hypothesize this improvement to less variation in point coordinates within a sector since all sectors are first transformed to a canonical coordinate frame before processing. This suggests that simulating streaming lidars can also serve as augmentation technique for full-sweep detection.
Diagnosis of Previous Streaming Methods We first analyze STROBE's low performance in Tab. I While all other methods aggregate points from past 10 sweeps after ego-mot mootooonon compensation (Point Warp). STROBE processes the points one sweep at a time, and aggregates information from the past sweeps by first transforming the features based on ego-motion (Feature Warp) and then fusing them with the current frame. As shown in the full-sweep case in Tab all the metrics for STROBE are significantly lower than other methods. We thus find Feature Warp inferior to Point Warp for detection and especially for velocity estimation. The average velocity error (3VE) of STROBE is 0.607 m/s, significantly higher compared to Cartesian Pillars with Point Warp (0.358 m/s). We speculate that this high velocity error is because the feature maps in STROBE don't 't encode time information on account of processing one sweep at a time as compared the other methods which encode the time lag for each accumulated point from the past 10 sweeps. For 1, 2, 4 sectors, STROBE enjoys the lowest latency because it processes fewer points compared to other methods, also shown in Fig. [ The pillar feature encoder runs faster. But this advantage disappears for more than sectors because there are also only smaller number of points processed by all other methods.
Compared to baseline Cartesian Pillars, the method of Han et al. only start to work when more than 8 sectors per sweep. For and sectors. even hurts the accuracy especially in object detection. This can be explained in Fig. 4 For 2 and 4 sectors, the feature map is fully occupied. Adding the pooled features from preceding sectors is like adding noise current sector, resulting in worse accuracy. Starting from 8 sectors, there is an empty region in current sector so adding the pooled features from preceding sectors is like padding the empty region, and therefore enlarging the context.
How Streaming Models Enlarge Context We further hypothesize not only our method and Han et work by padding context from preceding sectors, but also STROBE works by padding. As shown in Fig. 4 for 2 and sectors when the feature map is fully occupied, fusing features from previous sweep is like densifying the features. Starting from 8 sectors, when there is an empty region, fusing features is like padding the empty region. We argue that all existing streaming methods work by padding, but in different format. STROBE and Han et al. are restricted by the shape of the sectors and require empty region as placeholder, and the padded features Our method pads along the edges of feature maps and is not constrained by the shape of the sector.
added or fused to the placeholder.
Further Thoughts about Context We argue that context has two aspects. First is its feature values, for the texture information it carries. Second is its spatial relation to the object of interest. Since convolution is canslation-invariane convolutional neural networks alone do not encode spatial relation. The spatial relation is maintained in the spatial arrangement of neurons on the feature map. The stateful-RNN in Han al. must work together with the empty region as placeholder to maintain the spatial arrangement, while stateful- -NNN alone does not encode spatial relation. On the other
8

---

Detection
Semantic Segmentation
Panoptic Segmentation
30
Segmentee miuu
55
Panoptic Quality
Detection
PMT
74
R54
43
64
39
70
Runtime (H2)
Runtime (H2)
Runtime (H2)
Figure 5: Comparison of methods, full-sweep PolarStream (PLSI) and PolarStream with Bidirectional Context Padding using four sectors (PLS4), and other methods on the nuScenes dataset. We compare detection and semantic segmentation the nuScenes benchmark, and panop- tie segmentation on the nuScenes val split. We only compare with methods that report both ac- euracy and runtime. The methods in comparison CCVC, PointPainting (PNT),, Cylinder3D 37 (C3D), PolarNet 36 PPLN) SalsaNext ( CCBGSA4T Po- larNet+CBGS(P+C) and PaP 40 (PaP). We color our methods in green and other methods in red.
CenterPoint(33]CC (O)) AF2S3Net
Table 2: Comparison with state-of-the-ar methods on nuScenes Val split.
3DD Detection
Semantic Segmentation
Method
mAP
runtime(Hz)
#parameters(MIB)
mAP
runtime(Hz)
#paramcterx(MB)
CenterPoint 35
56.4
11
149
Cylinder3D 37
76.1
215
Ours-PI-S1-heave
56.2
11
149
76.8
1449
hand, although our padding along the feature map edges seems simple, is an effective solution both add feature values and maintain spatial relation of context.
5.4 Comparison with other Full-Sweep Models
As the full-sweep 3D object detection and LiDAR semantic segmentation have longer histories compared to streaming models, there are more full-sweep methods in the literature. We also compare with these methods. We present the results of our full- sweep PolarStream model with (PLSI) and best performing PolarStream model with n 4 (PLS4), with the same backbone as in PointPillars(I6) As shown in Fig. [5] our method maintains a good balance of runtime and accuracy compared to other methods on both the nuScenes detection and semantic segmentation benchmark. We achieve even faster runtime with PI.$4 while preserving almost same accuracy as PLSS. The panoptic segmentation results on the nuScenes val split show that our methods outperform all existing methods in PQ with at least 55% less runtime.
We also adopt heavier 3D ResNet [4] backbone as in CBGS 41 and compare with the state of
the methods for 3D object detection and semantic segmentation in Tab. 2 Our PhSl-heavy is able to match/beat the state-of-the-ae models for detection (CenterPoint (3S]) and segmentation (Cylinder3D(37) on the nuScenes validation set. In this work, focus on onboard applications so we only choose the same backbone as in PointPillars [6] for streaming.
5.5 Ablation Studies
The Effeet Multi-Scale Context Padding As shown in Tab the advantage of Multi-Scale Context Padding starts to show up for 8, 16 and 32 sectors, especially in segmentation. For 32 sectors, when the spatial view the most restricted, we observe the largest gain in detection mAP and segmentation mloU. Multi-Scale Context Padding does not improve or hurt the baseline polar pillars model for and 4 sectors, because the network already sees enough spatial view. As we increase the amount of context, from trailing-edge padding to bidirectional padding, observe

---

Table Ablation Study of Multi-Scale Context Padding on nuScenes Val split. CP: context padding; CP xl: tralling-edge padding: CP x2: bidirectional padding.
Panoptic Quality (PQ)
Segmentation mloU
Method
4
16
32
2
16
Polar Pillars
68.7
68.6
67.7
66.2
63.9
60.3
73.4
73.4
72.5
70.8
69.5
67.1
Ours-w/ CP
68.6
69
68.1
66.4
63.4
73.3
73.7
72.6
71.6
70
Ours-w/ CP x2
68.8
69.6
68.8
69.2
68.3
73.5
74.2
73.4
73.8
73.1
Detection mAP
Detection NDS
Method
2
4
16
32
2
4
16
32
Polar Pillars
51,2
52.1
51.9
52.5
51.9
46.7
60
61.4
61.4
60.9
60.3
55.8
Ours-w/ CP x1
52.2
53.6
52.4
52.3
49.3
60.3
61.2
60.8
60.4
58.8
Ours-w/ CPx2
51.5
53.2
52.7
53.9
52.4
60.3
61.2
60.9
61.4
more improvements. Surprisingly, we observe that with 2, 4, 8 and 16 sectors, our PolarStream with bidirectional padding even outperforms our full-sweep baseline of polar pillars in all metrics including PQ. detection mAP, NDS and segmentation IoU. suggests that streaming models can be both faster and more accurate.
The effect of Feature Undistortion and Range Stratified Convolution& Normalization We do the ablation studies with n = 1, i.e., the full-sweep case. To show how we close the gap of detection accuracy between polar pillars and Cartesian Pillars, we also list the results of Cartesian pillars with the same architecture and input size. We find that polar pillars outperforms Cartesian pillars in semantic segmentation mIoU (73.2 vs 72.1). which is also found in prior arts because points in the same polar pillar have less disagreement in the semantic label compared to those in Cartesian pillar. However, polar pillars less accurate in object detection due to the challenges we discussed. In Tab. 4 we show either Range Stratified Convolution& Normalization or Feature Undistortion helps to improve detection accuracy based on polar pillars (by 0.9 and 0.4 mAP respectively). With both techniques combined, we improve detection mAp from 48.2 2 to 50.3, narrowing the gap compared to Cartesian pillars (50.6). We also apply both techniques to Cartesian pillars and they do not improve Cartesian pillars, showing they only address the specific challenges of polar pillars, instead of improving the performance by adding more parameters to the network. Our techniques do not add noticeable runtime (0.5ms). In addition, trained with semantic segmentation. The improvement is more significant for Cartesian pillars, but Cartesian pillars suffer from slight drop in segmentation mloU.
find detection mAP can be improved when simultaneouly
Table 4: Ablation Studies on the validation split of nuScenes.
Method
seg
det
Stratified Norm&Comv
Feature Undistortion
Runtime (ms)
Det mAP
Seg mloU
29.5
73.2
Polar Pillars
38,2
48.2
38.4
49.1
38.4
48.6
38.7
50.3
44,9
51.2
73.4
29.8
72.5
Cartesian Pillars
38.1
50.6
38.5
50.6
44.5
52.3
72.1
Conclusion
In this work we propose a streaming model for simultaneous 3D object Detection, Lidar Segmentation and Panoptic Segmentation. Polar pillars is introduced as more compact representation for lidar sectors compared to previous methods. Multi-scale context padding including trailing-edge padding and bidirectional padding proposed to enhance spatial context of the streaming model with minimal latency. Additionally we make several improvements, Feature Undistortion and Range Stratified Convolution& Normalization, to address the problem of applying convolutions on polar grid. Our model showed significant improvements over previous streaming methods with lower latency.
10

---

References
[1] Alsfasser, M., Siegemund, J., Kurian, J., Kummert, A.: Exploiting polar grid structure and object shadows for fast object detection in point clouds. In: Twelfth International Conference on Machine Vision (ICMV 2019). vol. 11433, 114330G. International Society for Optics and Photonics (2020)
[2] Berman, M., Triki, A.R., Blaschko, M.B.: The lovász-softmax loss: tractable surrogate for the optimization of the intersection-over-umm measure in neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413 4421 (2018)
[3] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.1 1027 (2019)
[4] Casas, S., Luo, W., Urtasun, R.: Intentnet: Learning to predict intention from raw sensor data. In: Proceedings of The 2nd Conference on Robot Learning. pp. 947-956 (2018)
[5] Chen, Q.. Sun, L.. Cheung, E., Yuille, A.L.: Every view counts: Cross-view consistency in 3d object detection with Processing Systems (2020)
voxelization. Advances in Neural Information
[6] Chen, Q.. Sun, L.,, Wang, Z., Jia, K., Yuille, A.: Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots. In: European Conference on Computer Vision. pp. 68-84. Springer (2020)
[7] Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for au- tonomous driving. In: CVPR (2017)
[8] Cheng, R., Razani, R., Taghavi, E., Li, E., Liu, B.: 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. arXiv preprint arXiv:2102.04530 (2021)
[9] Cortinhal, T., Tzelepis, G., Aksoy, E.E.: Salsanext: Fast, uncertainty-aware semantic segmenta- tion of lidar point clouds for autonomous driving. arXiv preprint #XXv:2003.00655 (2020)
[10] Fan, L., Xiong, X., Wang, F., Wang, N., Zhang, Z.: Rangedet: In defense of range view for lidar-based 3d object detection. arXiv preprint utXiv:2103.1000 (2021)
[11] Frossard, D., Suo, S., Casas, S., Tu, J.. Hu, R., Urtasun, R.: Strobe: Streaming object detection from lidar packets. arXiv preprint arXiv:2011 1.06425 (2020)
[12] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11), 1231-1237 (2013)
[13] Han, W., Zhang, Z., Caine, B., Yang, B., Sprunk, C., Alsharif, O., Ngiam. J., Vasudevan, V., Shlens, J., Chen, Z.: Streaming object detection for 3-d point clouds. In: European Conference on Computer Vision. pp. 423-441 Springer (2020)
[14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[15] loffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference machine learning. pp. 448-456 PMLR (2015)
[16] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang. J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 12697-12705 (2019)
[17] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, Focal loss for dense object detection. In: Proceedings of the IEEE international conference computer vision. pp. 2980-2988 (2017)

---

[18] Liu, W., Anguelov, D.. Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European conference on computer vision. pp. 21-37. Springer (2016)
[19] Luo, W., Yang, B., Urtasun, R.: Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with single convolutional net. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
[20] Maturana, D., Scherer, S.: Voxnet: 3d convolutional neural network for real-time object
recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 922-928. IEEE (2015)
[21] Meyer, G.P., Laddha, A., Kee, E., Vallespi- Benlespi-Gnzzaez,, C., Wellington, C.K.: Lasernet: An effi- cient probabilistic object detector for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12677-12686 (2019)
[22] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652-660 (2017)
[23] Rapoport-Laave,, M., Raviv, D.: It's all around you: Range-guided cylindrical network for 3d object detection. arXiv preprint arXiv:2012.0312) (2020)
[24] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cmn: towards real-tim object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence (9()). 1137-1149 (2016)
[25] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. pp. 234-241. Springer (2015)
[26] Simon, M., Milz, S., Amende, K., Gross, H.M.: Complex-yolo- Real-time 3d object detection on point clouds. CoRR (2018)
[27] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo. J., Zhou, Y., Chai, Y., Caine, B., al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2446 2254 (2020)
[28] Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., Han, S.: Searching efficient 3d architec- tures with sparse point-voxel convolution. In: European Conference Computer Vision. pp. 685-702. Springer (2020)
[29] Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4604-6611 (2020)
[30] Yan, Y.. Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
[31] Yang. B., Liang, M., Urtasun, R.: HDNET: Exploiting HD maps for 3d object detection. In: CoRL (2018)
[32] Yang, B., Luo, W., Urtasun, R.: Pixor: Real-time 3d object detection from point clouds.
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 7652-7660 (2018)
[33] Yang, B., Luo, W., Urtasun, R.: PIXOR: Real-time 3d object detection from point clouds. In: CVPR (2018)
[34] Ye, Y., Chen, H., Zhang, C., Hao, X., Zhang, Z.: Sarpnet: Shape attention regional proposal
network for lidar-based 3d object detection. Neurocomputing 379, 53-63 (2020)
[35] Yin, T., Zhou, X., Krähenbühl, Center-based 3d object detection and tracking. arXiv preprint arXiv:2006.1 1275 (2020)
12

---

[36] Zhang, Y., Zhou, Z., David, P., Yue. X., Xi. Z., Foroosh, H.: Polaret: An improved grid repre- sentation for online lidar point clouds semantic segmentation. arXiv preprint arXiv:2003. 14032 (2020)
[37] Zhou, H., Zhu, X., Song, X., Ma, Y., Wang, Z., Li, H., Lin, D.: Cylinder3d: An effective 3d framework for driving- scene lidar semantic segmentation. arXiv preprint arXiv:2008.01550 (2020)
[38] Zhou, Y., Sun, P., Zhang, Y., Anguelov, D., Gao, J., Ouyang, T., Guo, J., Ngiam, J., Vasudevan, V: End-to-end multi-view fusion for 3d object detection in lidar point clouds, arXiv preprint 1rXiv:1910.06528 (2019)
[39] Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4490 4499 (2018)
[40] Zhou, Z.., Zhang, Y., Foroosh, H.: Panoptic-polarnet Proposal-free lidar point cloud panoptic segmentation. arXiv preprint arXiv:2103.1496. (2021)
[41] Zhu, B., Jiang, Z., Zhou, X., Li. Z., Yu, G.: Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint 19X8.0900000999 (2019)
13


Reviews:
[{"review_id": "jEjK-JamuIg", "reviewer": "Reviewer_X4zp", "summary": "In this manuscript, the authors propose streaming based LiDAR detection using polar grid representation. Incorporating dedicated convolution and undistortion methods, the proposed method shows good performance on the NuScenes dataset.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main issue of this manuscript is lack of clarity and analysis which makes the claimed contribution unclear:\n* In figure 1, the authors claim polar representation is better than cartesian in terms of memory and computation. However, no such comparison is provided\n* There is overall no result visualization or case analyses (segmentation+detection) to demonstrate the usefulness of the proposed method.\n* For table 1, the detection result for <=4 sectors are worse than Cartesian, even with CP. Why is this the case? This seems to contradict with the claim that Polar representation is better. And why segmentation tasks do not have such an effect? Insights are needed.\n* Range Stratified Convolution: this is extremely unclear for how the kernels are allocated for each grid. Also relevant ablation studies are needed.\n* Section 5.2, multi-scale padding: It is still unclear why detection performance improves when the number of sectors increases. It might be because there are more overlapping bbox proposals, or NMS strategy. More detailed analysis is needed.\n\nIn addition, only running the experiment on the NuScenes dataset is not sufficient in my opinion. I suggest the authors add the Waymo dataset. Also directly accumulating 10 frames may incur localization error which may interfere with the detection result. Single frame baseline is also needed.\n\nSome other issues:\n* Not sure why the authors put emphasis on the point pillar backbone. Any other backbone can do the job.\n* For feature undistortion, why only apply this method on the classification head? The more appropriate way is to apply on the backbone. This experiment is needed.\n* Section 5.3, diagnosis part: this part of analyzing two previous methods is not relevant to the main body. \n\nEvaluation\nOriginality: the method is original\nQuality: the proposed method and experiments are technically sound.\nClarity: main issue is the clarity, mentioned in my comments\nSignificance: the low latency LiDAR detection model is of interest to broad audience in the community.\n\n\n================post rebuttal====================\n\nI would like to thank the authors for the detailed reply. While the reply addressed some of my concerns, I am still not convinced by 1) the memory and computation advantage, and 2) the performance gain vs. the Cartesian coordinate (e.g., only larger sectors work, but sacrifices the overall performance). Also the explanation of the solved cases and Range Stratified Convolution is still vague. My score remains the same.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Z1qrdCKURyf", "reviewer": "Reviewer_1vpo", "summary": "The paper proposes some modifications to the work of [11,13], which processes 3D lidar data by analysing consecutive sectors of a 360 deg. scan as soon as they are acquired. In particular, it suggests using polar coordinates rather than cartesian as a way to more efficiently adapt to the sectors, which are circular portions. This reduces the memory required to process the lidar data within a neural network, but requires specific additional stages in the neural network itself to take care of the different shape and coordinate system of the input data. The authors hence propose specific stages to take care of this, namely feature undistortion and range stratified convolution and normalization. Results are ablated and compared against [11,13] + a cartesian-based equivalent of their method on the NuScenes benchmark.  ", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper tackles an important problem in the current literature around applications of ML to 3D computer vision, i.e. learning from 3D lidar data. The two methods that this work aims to improve are quite recent (2020) and to the best of my knowledge represent the state of the art  for sequentially processing portions of an entire lidar scan. \n\nOne limitation of this work is represented by the overall novelty of the method. Since the idea is to provide a representation of the input data that better fits the circular portions wrt. [11,13], this has an incremental nature wrt. the work in [11,13], since it can be regarded as an optimization of that idea. Also the idea of undistorting feature to let a NN process a non-cartesian input representation is not novel but shares similarities with previous work [a,b]\n\nThe results provided in Table 1 show the usefulness of the proposed approach against [11,13], and the authors also provide a nice discussion trying to explain the reasons for this superiority in Section 5.3. Nevertheless, extending Table 1 to include a comparison against the state of the art would have been useful to properly position the performance of the proposed approach with respect to the current literature. The results in Figure 5 go a bit in this direction and, from the provided charts, it seems like the current approach is still behind in terms of detection and semantic segmentation, while reports state of the art results in panoptic segmentation. \n\nThe paper is sometimes hard to read, due to the presence of typos as well as sentences that contains mistakes or are simply hard to understand. To improve the readability, especially around the methodology explanation, a careful review of the language used would be beneficial. \n\nTaking everything into account I believe the paper falls short of requirements for acceptance especially with regards to the novelty and presentation aspects. \n\nAdditional points:\n- line 118: point are accumulated from 10 successive frames: how did the author pick this specific parameter, and what is the weight of this value on the performance?  \n- line 181: the Range Stratified Normalization normalizes over individual regions within a certain range rather than on the entire spatial domain: how are these regions selected? It seems to me like the choice/number of regions should be triggered on the distance of objects and scene components with respect to the sensor. As these regions seem to be obtained by discretizing the spatial range, what happens to an object lying in between two regions? Would it receive two different normalizations?\n\nReferences\n[a] Yu-Chuan et al, Flat2sphere: Learning spherical convolution for fast features from 360 imagery. In: Proc. Conf. on Neural Information Processing Systems (NIPS) (2017)\n[b] Coors et al, SphereNet: Learning Spherical Representations for Detection and Classification in Omnidirectional Images, ECCV 2018.\n\n\n---\nPost rebuttal comment:\n\nAfter going through the rebuttal and the comments from the other reviewers I feel my initial thoughts on the paper are confirmed. The method looks at an important problem but falls short in terms of novelty, clarity and experiments. In particular I share the concerns with reviewer X4zp around the justification of polar coordinates stemming from the results presented in the paper, as well the readability issues that are raised across all reviewers. On a more minor note, the authors acknowledged in their rebuttal the lack of a more thorough review of related work around weight undistortion relatively to the citations that I suggested. While I agree the method is substantially different from those in the literature, a proper reference to the state of the art in this particular field is important to bring the proposed contribution into context. I am thus confirming my initial rating.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I have no specific concerns in these regards. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "WhjKHwn1mWq", "reviewer": "Reviewer_dNFe", "summary": "This paper studies streaming algorithms for low-latency LiDAR object detection and segmentation. The authors propose to use the polar coordinate system to represent the point cloud data in order to reduce memory and compute waste. They further introduce two techniques to bridge the performance gap between polar and cartesian-based solutions: (1) multi-scale padding from neighboring sectors to increase the spatial context, and (2) feature undistortion and range stratified convolutions to address the distortion. The authors have evaluated their proposed model on the nuScenes dataset and have achieved good empirical performance.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is well-written and easy to follow. The proposed streaming LiDAR processing framework is technically sound and has achieved relatively good empirical results.\n\nMy primary concern of this paper is the justification for using polar coordinates. The authors have argued in the introduction (L27-L30) that using cartesian coordinates (and cuboid-shaped voxels) will waste both computation and memory. However, this does not seem to be the case from the experimental results (Table 2 and Figure 1): the cartesian-based model has almost the same measured latency as the polar-based model. At the same time, the polar-based model has inferior detection performance (1-2% in mAP) compared with the conventional cartesian-based model. Therefore, the authors should explain their motivation behind switching to polar coordinates in their rebuttal.\n\nThe superior performance on the two segmentation tasks is not very surprising since similar observations have been made in Cylinder3D [Zhu et al., CVPR 2021]. Besides, the semantic segmentation results presented in this paper are more than 5% lower than the state-of-the-art submissions on the nuScenes leaderboard. Thus, it is unclear whether their proposed solution is still effective when applied to these models with better performance.\n\nAmong the three downstream tasks, object detection is more important than the other two since it is usually deployed in the perception stack, while semantic/instance segmentation is usually used offline to build HD maps (where the latency does not matter as much). The authors should justify why they choose to perform these tasks jointly and whether this joint detection-segmentation task is a valid setting.\n\nFinally, the authors should double-check their \"et al.\" usages throughout the manuscript:\n* L86, L94, and L269: missing \".\".\n* L236: misspelling as \"el al\".\n\nThere should also be a space between \"et al.\" and the citation.\n\n**Post-Rebuttal Comments:**\nThe authors' rebuttal has not addressed my major concern (i.e., the justification for using polar coordinates). As the use of a very powerful GPU is the main reason for the negligible speedup, it would be necessary to deploy these models on lightweight edge devices (such as Jetson AGX Xavier) to verify that the polar-based models are indeed faster than the cartesian-based models under a more realistic hardware setup. Without these results, it is unclear whether polar-based models are superior in the efficiency-accuracy tradeoff. Therefore, I will keep my original score and vote for rejecting this paper.\n\nReferences:\n* Zhu et al., \"Cylindrical and Asymmetrical 3D Convolution Networks for LiDAR Segmentation\", CVPR 2021.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "This paper does not have noticeable negative societal impacts. As for limitations, their proposed solution does not seem to work well for LiDAR detection, which is arguably much more important than the two segmentation tasks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a method for object detection and segmentation in LIDAR data. The reviewers generally appreciated the importance of the problem and the efforts made in the paper. However, several issues where raised on novelty, writing and experiments.

The authors' response could not address the most critical concern, which was the justification of polar coordinates. The reviewers judged that this claim was not backed up by the experiments, and a consensus for rejection emerged.

I concur.
Decision: Accept (Poster)