PolarStream: Streaming Object Detection and Segmentation with Polar Pillars
Authors: Qi Chen
Sourabh Vora
Oscar Beijbom
Venue: NeurIPS 2021 Poster (2021)
Keywords: Streaming Object Detection
3D Object Detection
LiDAR Segmentation
3D Panoptic Segmentation
Polar Grid

Abstract: Recent works recognized lidars as an inherently streaming data source and showed that the end-to-end latency of lidar perception models can be reduced significantly by operating on wedge-shaped point cloud sectors rather then the full point cloud. However, due to use of cartesian coordinate systems these methods  represent the sectors as rectangular regions, wasting memory and compute. In this work we propose using a polar coordinate system and make two key improvements on this design. First, we increase the spatial context by using multi-scale padding from neighboring sectors: preceding sector from the current scan and/or the following sector from the past scan. Second, we improve the core polar convolutional architecture by introducing feature undistortion and range stratified convolutions. Experimental results on the nuScenes dataset show significant improvements over other streaming based methods. We also achieve comparable results to existing non-streaming methods but with lower latencies.

PolarStream: Streaming Lidar Object Detection and Segmentation with Polar Pillars
Qi Chen'
Sourabh Vora Motional Santa Monica, CA 90401
Johns Hopkins University Baltimore, MD 21218 gchen420jhu. edu
sourabh. vora@motional. com
Oscar Beijbom Motional
Santa Monica, CA 90401
oscar bei wwwp:oootiiooooomm com
Abstract
Recent works recognized lidars inherently streaming data source and showed that the end-to-end latency of lidar perception models can be reduced significantly sedge-shaped point cloud sectors rather then the full point cloud. by operating However, due to use of cartesian coordinate systems these methods represent the sectors rectangular regions, wasting memory and compute. In this work we propose using polar coordinate system and make two key improvements on this design. First, we increase the spatial context by using multi-scale padding from neighboring sectors: preceding sector from the current scan and/or the following sector from the past scan. Second, improve the core polar convolutional architecture by introducing feature undistortion and range stratified convolutions. Experimental results on the nuScenes dataset show significant improvements over other streaming based methods. We also achieve comparable results to existing non-streaming methods but with lower latencies.
Introduction
The ability to accurately perceive objects in dense urban environments still remains challenging problem for self-driving cars. While such self-driving cars typically deploy wide variety of sensors lidars play a key role due to the accurate range information provided. Driven in part by the availability
271. the last decade has seen tremendous progress in lidar based 3D 10]. However, these methods all ignore the fact that most lidar sensors scan the scene sequentially as the lidar rotates around the 7-axis. They instead wait for the rotational scan to complete (colloquially known as full sweep) before processing data, thereby introducing large data capture latency (usually 50 100 ms).
of benchmark datasets 22
3
object detection 39
16
32
21
First, Han et al. 133 and then STROBE [1 recognized this problem and proposed solutions which processed lidar sectors (shown in Fig. M soon as they arrived. They showed that streaming based architecture can achieve significantly reduced latency over the traditional non-streaming baselines. Both of these methods encode the point clouds as an image in bird's s-eye view (BEV) using cuboid- shaped voxels. In doing so, they ignore the natural polar representation formed by the lidar sectors. Using cuboid-shaper voxels restricts them to performing convolutions on the minimal rectangular region enclosing the point cloud sector which wastes both computation and memory. As shown in Fig.
large portion of the enclosed rectangular region remains empty.
"work done while interning at Motional
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

700
Previous work
67.5
Panoptic Quality
65.0
62.5
57.3
Han et STROSE
Ours
52.
PolorStream
30 60
90
End-to-End Latency / ms
Figure 1: Left: An illustration of streaming lidar point clouds bird's eye view. Lidar point clouds arrive as wedge-shape sectors (shown in gray masks) as the scanner rotates. Previous methods, Han al. [33 and STROBE(L], represent the sectors using rectangular regions, wasting half of memory and computation for empty regions. Ours represents the sectors as wedge-shape regions using a polar grid. Right: Comparison of different streaming methods wrt. Panoptic Quality vs End-to-En End Lantency as we slice the full sweep into n = 2, 4, 8, 16, 32 sectors using the NuScenes val split. The end-to-end latency includes 50/n ms for LiDAR scan and the total runtime of the algorithms.
Another challenge associated with streaming perception models the limited view of the scene observed by each sector. Objects close to the ego- ago-vehicle can often be fragmented across multiple sectors as shown by the car highlighted in green in Fig Han et al. [3] proposes to increase the context available to the model by maintaining recurrent memory across consecutive sectors. STROBE [ also aggregates representations from the previous sectors by maintaining full-sweep feature maps across multiple scales. However, both these solutions add extra computation.
In this work, we propose to encode individual point cloud sectors using polar pillars. Polar pillars naturally address the inefficiency of existing streaming approaches by representing the point cloud sectors as more compact wedge-shaped regions as shown in Fig. []] Further, we propose a simple Minimall-latenee approach to enhance the context available to the model by simply padding the representation of the neighboring sectors across multiple strides of the backbone. Using polar pillars allows us to pad features from the preceding sector of the current scan and/or the following sector from the previous scan, no matter how many sectors the full sweep divided into.
The polar BEV representation has recently started gaining attention in the lidar perception literature primarily because it balances the points across grid cells. In fact, polar grid outperforms the cartesian grid on the lidar segmentation task [36 37.. However, the detection peformance
polar grid still lags the cartesian grid [ 5 (23]. This is because of the distortion the objects undergo when this representation is ultimately unfolded to rectangular representation to enable the use of convolutional layers. The object represented by the green box in Fig. 2
shows example of this distortion. Further, the distortion increases with range as the pillars progressively become larger. This makes a polar representation compatible with the ianislatioonivvriiann property of convolution.
In this work, we propose several techniques to address the distortion problem described above. We first propose Feature Undistortion module which transforms the polar representation into canonical Cartesian representation (as shown in Fig. (2 for classification branch. Next, we propose using the Range Stratified detection head. These layers apply different convolution kernels and normalization based on range (Fig to cater the changing pillar sizes in a polar grid. Our proposed model closes the gap on 3D object detection models using cartesian representations without adding any significant latency.
layers on the regression branches of the
Finally, train multitasking streaming models that do simultaneous 3D object detection. lidar segmentation and panoptic segmentation, for the first time in literature. Results on the nuScenes dataset show that our proposed model PolarStream outperforms all streaming methods in both panoptic quality and speed. PolarStream also stays competitive with the top-performing lidar perception methods the nuScenes leaderboard while being least twice fast as the rest. We do several ablation studies and extensive analysis to show the effectiveness of PolarStream.
In summary, contributions are:

---

An efficient streaming based lidar perception models using polar grid.
Multi-scale context padding: an efficient approach to enhance the context of streaming lidar perception models
Several improvements to the core problem of applying convolutions on a polar grid: Feature Undistortion, Range Stratified Convolution&Normalizattt all add minimal lateney to our model.
2 Related Works
2.1 Non-streaming lidar perception
Most lidar perception architectures take inspiration from the image perception literature 224
18 16 32
17
Some single-stage methods typically convert the point cloud into bird bird's-eye view image 33
or range view image (21 [O] and perform detection in those views. The most common paradigm is to convert the lidar point cloud into a BEV image as offers several advantages like a lack of scale ambiguity, near lack of occlusion, the ease of fusing HD maps [31] and performing simultaneous
detection and trajectory predictions [4] [9]. To convert the point clouds into a BEV representation, most existing models choose group the points into voxels. The most commonly used voxels
are cuboid-shaped based on Cartesian coordinates. Vox (20). MV3D (7]. Pixor (33, Complex- YOLO (26) represent the saboid-shaped voxels as occupancy grids. To avoid quantization effects of occupancy grids and extract richer voxel features, VoxeINet [39] samples a fixed number of points within each voxel and applies simple PointNet [22] them. For efficiency, PointPillars [T6] discretizes the 3D space into pillars so there is only one voxel along the height dimension.
Some recent methods that operate on BEV start to explore polar voxels for point clouds. For 3D) object detection, Alsfasser et al []] voxelizes points under the Cylindrical Coordinate System, MVF 38 adopts both cuboid-shaped voxels and spherical voxels, and CVCNet combines cylindrical and spherical coordinate system into one Hlybrid-Cylindriial-SSheee
(HCS) coordinate system to detect object from both bird's 's eye view and range view. On the other hand, the success of PolarNet [36] and Cylinder3D (37] shows the advantage of Cylindrical grids over Cartesian voxels in LiDAR semantic segmentation. Panoptic-Polate PolarNet 40) further extends PolarNet to the task of panoptic segmentation.
2.2 Streaming lidar perception
Streaming lidar perception is relatively new in literature and offers compelling argument in reducing the end-to-end latency. Han et al [13] proposed a couple of enhancements to convert a 3D object detector to operate streaming data: a) using an LSTM to accumulate features from preceding sectors and b) applying stateful NMS to suppress objects across multiple sectors. STROBE [T] accumulates features not only from the preceding sectors of the same scan but also from the previous scan by maintaining multi-seale memory feature maps. Features extracted from the current sector is concatenated and fused with the corresponding cropped region in the memory feature maps.
PolarStream
In this section, we introduce PolarStream, streaming model based on polar pillars. We introduce how we prepare lidar streaming data in Sec3.1] polar pillars as representation for point clouds sectors in See 3.2 the simultaneous detection and segmentation model including techniques to improve detection on a polar grid in 3.3 3.3 and multi-scale context padding to enlarge context in Sec 3.4 3.1 Streaming LiDAR Inputs
Since there is no streaming lidar dataset available, we simulate a streaming system from the NuScenes dataset [3] by slicing the point clouds into sectors according to their azimuth. As shown in Fig [ each sector is like a slice of a full pizza. We try n = 1, 1,2, 4, 8, 16, 32 sectors in our experiments, where = means full sweep. The dataset contains 1, 000 scenes, comprising 700 scenes for training, 150 scenes for validation and 150 scenes for test. Each scene is of 20s duration, captured by 32-beam lidar. 40, 000 frames are annotated in total, including 10 object categories such cars, motorcycles and pedestrians and six stuff classes such as vegetation and drivable region. We consider 10 object classes for detection, 16 classes in total for semantic and panoptic segmentation.
3.2 Polar Pillars
The point clouds sector consists of N points, each represented by a vector of point feature f, (r, Op Zp, zp:Tps Tp Ups ip. (p), where (xp Up, zp) is its Cartesian coordinates. (rp, is the polar coordinates. i, is the reflection intensity and lp is the timestamp when the lidar point is captured. Points are accumulated from 10 successive frames in total to obtain denser point clouds. The points from previous frames are compenstem and transformed to current frame. We group the
3

---

Clouds Poler Pilaars
polar grid points
Classification Head
Interpolated cartesian grid points
Pillar Feature Encoder
Backbone (2D CNN)
3D Box
Regression
3D
Heads
Feature Undistortion
Unfold in BEV
Upsample
Concat
separate kernels
R
Segmentation
Head
50
-202
Range Straiified Comvoluionn&Nommllzaiioo
Figure 2: Simultaneous LiDAR object detection and segmentation network with polar pillars. We adopt the same backbone as in PointPillars[16 [16], and add a semantic segmentation head in parallel with the detection heads. The input wedge-shape pillars are unfolded into rectangular feature map for convolution. The object (green box) is distorted because one end near the sensor looks bigger and the other end far from the sensor looks smaller. Feature Undistortion is applied to classification head to mimic bilinear sampling and interpolate cartesian pillar features from polar pillar features. Range Stratified Convolution& Normalization is applied to center offset regression head.
points according to the cylindrical pillar resolution (òr, 80, 6z) where d2 Zmax Zmain SO there is only one pillar along the height dimension. Following MVF [8). we adopt dynamic voxelization to sample all points within each pillar, instead of randomly sampling a fixed number of points per pillar.
3.3 Simultaneous Detection and Segmentation
We design PolarStream: a simultaneous object detection and segmentation network by extending PointPillars [66], one of the most widely used 3D object detectors balancing accuracy and speed. As shown in Fig 2 PolarStream consists of Pillar Feature Encoder, followed by a 2D CNN backbone
and U-Net 25 like structure. On top are the detection and segmentation heads.
Detection Heads We adopt CenterPoint (35 heads with modifications make it compatible with polar pillars. To assign targets to the 10-class heatmap to indicate the objects, the gaussian radius of the object center is computed using the span of range and azimuth of the object bounding box, instead of using length and width of the box. Following CenterPoint, we also regress the center offset as da, dly the bounding box size 1, w., h as log 1, log w. log h. and predict the bounding box height z. We regress the relative bounding box orientation Φ as cos σ, sin ø and relative velocity as uz Uy similar to (23). Unlike most methods, which use multi-group detection heads that partition object classes to several groups according to their size, we use singlc-grous group detection heads to balance accuracy and speed. A comparison against multi-group detection heads shown in Supplementary. For streaming data with n > 1, we apply stateful-NMS proposed in Han et al. [3]
Segmentation Head To extend PointPillars for segmentation, we add semantic segmentation head in parallel with the detection heads. The segmentation head is made of a single 1x convolution layer. The input for the segmentation head is concatenation of the outputs from pillar feature encoder and bilinearly upsampled features from the 2D backbone.
Panoptic Fusion Similar to Panoptic- Polaanet [40], for each point belonging to things, we predict the instance id as the box id whose category is the same and center is the nearest. For streaming data with > 1, the panoptic segmentation task is not well defined. For example, the points in the det sector may belong to the box in the (i + 1),A sector the majority of the box is in the (i + 1) sector. However, when are doing panoptic fusion for its. sector, we do not have information from the (i + 1), sector. Therefore we choose global panoptic fusion for streaming point clouds, i.c., we assign instance ids according to the boxes from all sectors of the same sweep.
Multi- Learning We adopt Focal Loss [1 for classification and LI loss for bounding box regression, orientation and velocity estimation. For segmentation, use the weighted cross-entre cross-entropy loss and lovasz-softmax loss 2] The total loss is the weighted sum of losses for each component.
Feature Undistortion As mentioned in See objects have distorted appearances with polar pillars, we propose Feature Undistortion to undistort the features. As shown on the top right of Fig the idea of undistortion is to interpolate features at cartesian pillar locations from the original polar pillar

---

locations so that the translation translaton-invariint property of convolution applies. We find the connection of bilinear sampling to convolution and mimic bilinear sampling using convolution. For bilinear sampling, the interpolated features at point p can be sampled from its neighboring points.Vp
Jp
Wk Ph
PEENP
where wi is a function of distance(p. pk).
We find Equutioon]aas the similar form to convolution, except that for convolution wi is fixed because same kernel is slided through every location of the feature map. To make wy distance-dependent, we tweak Equation 1 by adding a new parameter wi so Equation can be rewriten as:
Sp
www.sSr
(2)
PEENP
where w/, is conditioned on distance(p, P&). We model w/ as the output of a neural network. We build standalone fully convolutional network g that takes position encodings at ps and its neighboring points Np i.e {pe, = (ra, cos 0, sin 0,, Zi, D; E Ny.. U Up J as input, and output wy.- Simply put:
w/ee = s((ee)))
To make it more general, we also add a bias term 1/g and another standalone network so that b/2 a( (pee]) and
fp Σ
wu(Wffn 6k)
PLEN
I and 9 is trained together with our main network, and during inference w/. and b/g are fixed for each location so it does not need extra runtime for and q. We apply feature undistortion in center heatmap prediction.
Range Stratified Convolution & Normalization Another challenge with polar pillars that the center offset is dependent on range and azimuth so it has different statistics at different regions: suppose the heatmap center is at (re, fz), and the target is at (rt. 0e). The center offset is
d₂ = Tı cos 0 - Te cos 0g dy = r; sin A/ re sin 6.
(5)
For simplicity, assume Tt- = re, ie. the center offset moves along a circle. Suppose 02 > Ag then 8 = 0g + 0, < 6g 80
(6)
where 0 is small angle and 80 is the polar pillar angle size. Then
cos 0.) & -re0s Og d₂ (cos - 0.) coo
range and azimuth and observe that for Cartesian pillars center offset ranges from 1 to 1 and mean is 0.49 and std 0.28, while polar pillars center offset ranges from -2 to 2, mean is 0 and std is 0.64. The polar std is much larger than that for Cartesian pillars. Hence it's more difficult to regress center offset based on polar pillars. Based on these observations, convolution and batch normalization 13). As shown on bottom right of Fig Range Stratified Convolution applies individual kernels different ranges and Range Stratified Normalization only normalizes over individual regions within certain range instead of entire spatial dimension. We apply Range Stratified Stratified Normalization to the shared convolution for detection heads.
Similarly, we can derive that d, is also dependent
propose Range Stratified Convolution& Normalization instead of regular
to center offset regression. We also apply Range
3.4 Multi-Scale Context Padding
Trailing-Edde Edge Context Padding As shown in Fig the sector unfolded to a rectangle feature map on r-0 plane as input for convolution. The lidar sectors arrive one after another by increasing the angle the sensor scans so the unfolded feature map of sector is spatially connected to its preceding sector along 0 dimension. This unique property of using polar pillars inspires to, instead of zero- padding along 0 dimension, pad the features from preceding sector where is spatially connected to current sector. The receptive field of neuron increases as the neural network goes from bottom layer to top and the network encodes multi-scale representation of the input at different stages. This motivates us to pad context from preceding sector before every convolution of the 2D CNN backbone,
as illustrated in trailing-edge padding of Fig Although we only pad a few columns to the feature map, the neural network is replenished with sufficient context from multiple ranges and multiple scales at different stages of the network. We keep sero-padding for r dimension and the other end of the 0 dimension, as the other end of 0 dimension points to the future sector.
5

---

Trailing-adge padding
Backbone (2D CNN)
1=2/8
1=0
t- 1/6
R
ped
ped
Bidirectional padding
merge
marge
1=0
t=1
warp
0
conv
t-7/6
R
deconv
pad
ped
Figure 3: Multi-Scale Context Padding. We present both trailing-edge padding and bidirectional padding. Trailing padding pads current sector with features from preceding sector. Bidirectional padding additionally pads current sector with features from 'following' sector of past time frame. Full-sweep feature maps merged for past time frame and warped the coordinate system of current time by ego-motion compensation. Context Padding is applied to every convolution in the backbone.
Bidirectional Context Padding With trailing-edge padding the current sector is padded with context from preceding sector. To provide further context we pad the leading-edge with warped features from the following sector of the previous sweep. To do this aggregate the full-sweep multi-scale feature maps from the previous sweep and warp the feature maps to the coordinate system of current sweep using ego-motion compensation. We then pad the leading edge of the current sector with the corresponding warped features spatially connected to the current sector.
4
Implementation
4.1 Network Details
For polar pillars with n sectors per sweep, r, 2, 2 range is 30.3),m [-3.1488, -3.1488
6.2976/n] rad and [-5,3]m m,, the pillar size is (0.098, 0.0123, 8). For Cartesian pillars, the pil- lar size (0.2, 0.2, 8). When n = 1, x, y, range leaving same input size of 512 512 1 for both Cartesian pillars and polar pillars when 1. We find the minimal rectangular region to enclose the sectors when n > for Cartesian Pillars. We set segmentation loss weight to 2 and classification loss for both polar pillars and Cartesian pillars. For Cartesian pillars the bounding box regression weight is 0.25. For polar pillars, since regression is harder, we the loss weight to 0.5. We make sure they the best configuration for each setting. For and in Feature Undistortion, they share the same architecture: a 3x3 conv followed by 1x1 conv with tanh as activation. We show the network architecture in Supplementary. All runtimes are measured on single V 100 GPU using Pytorch.
51.2, 51 .2mm, [-51.2,51 21 m and [-5, m
4.2 Augmentation
We adopt class- balanced sampling proposed in CBGS [1 sectors, we conduct random flipping along x, y axes, scaling with a scale factor sampled from [0.95, 1.05], rotation around z axis between [-0.3925, 0.3925] rad and translation in range [0.2, 0.2, 0.2) m x, y, 2 axis. Unlike most methods, we do not use database sampling 30 for fast training.
Before slicing the point clouds into
Experiments and Results
5.1 Evaluation
We gather the predictions from individual sectors and evaluate PolarStream similar to full-sweep meth- ods. We evaluate 3D detection and lidar semantic segmentation on the NuScenes benchmark [3. The detection mean average precision (mAP) based on the distance threshold (i.e. 0.5m, 1.0m, 2.0m and 4.0m). Additionally, nuScenes detection score (NDS) (3), weighted sum of mAP and

---

Table 1: Comparison of streaming methods on nuScenes Val split. CP: context padding; CP xl:
trailing-edge padding; CP x2: bidirectional padding.
Panoptic Quality (PQ)
Segmentation mloU
Method
4
16
2
16
Cartesian Pillars
67.8
66.8
67.6
65.5
64.1
59.8
72.1
71.5
70.9
70.2
68.6
65.7
Han al.
66.3
67.4
66.2
64.6
61.6
70.6
71.5
70.1
69.2
66.5
STROBE
63.8
65.2
66.2
65.2
62.9
59.5
69.2
69.7
69.8
69.6
67.6
65.6
Polar Pillars
68.7
68.6
67.7
66.2
63.9
60.3
73.4
73.4
72.5
70.8
69.5
67.1
Ours- PolaarSStrt
68.8
69.6
68.8
69.2
68.3
73.5
74.2
73.4
73.8
73.1
Detection mAP
Detection NDS
Method
4
16
2
16
Cartesian Pillars
52.3
51.3
54.9
52.4
47.6
60.7
60.3
62
57.9
59.5
57.1
Han al. 13
50.9
52.9
53.8
52.7
50.6
59.6
60.3
60.8
60.3
58
STROBE
46.9
48.7
49.4
47.9
45.4
42
53.8
54.6
51.5
48.9
47.1
44.8
Polar Pillars
51.2
52.1
51.9
52.5
51.9
46.7
60
61.4
61.4
60.9
60.3
55.8
Ours-Polle PolarSSoarSt
51.5
53.2
52.7
53.9
52.4
60.3
61.2
60.9
61.4
60
precision on box location, scale, orientation, velocity and attributes. For semantic segmentation, we follow the standard mean intersection-ovve--uiioo (mloU) metric. Since nuScenes does not provide instance labels for panoptic segmentation, we follow Panoptic-PolarNer [40] to generate labels and evaluate panoptic segmentation on validation split using the Panoptic Quality (PQ) metric.
5.2 Comparison with other Streaming Methods
Baselines Han et al. and STROBE([]] did not release their code and in addition performed evaluation on two different datasets. To enable benchmarking we re-implemented their methods using the same backbone and input resolution we re-implemented stateful-NMS and stateful-RNN STROBE. We did not implement the HDD map branch in STROBE in order ensure fair comparison. We also apply stateful-NMS and global panoptic fusion to all the methods in comparison as they just post-processing techniques. We extend both methods the task of simultaneous object detection, semantic segmentation and panoptic segmentation. We also provide baselines that simply apply Cartesian pillars or polar pillars individual point clouds sectors. We compare panoptic quality, segmentation mIoU, detection mAP, NDS with the baselines using n = 1, 2, 4, 8, 16, 32 sectors [T. We also show the comparison of our method Han et al. and STROBE wrt. PQ vs. end-to-end latency (Fig. I.
use and evaluated the nuScenes dataset. Specifically
Han el al. and multi-scale memory module in
Results Tab. shows that PolarStream outperforms all previous streaming methods, including the Cartesian pillars baseline, in both PQ and Segmentation mloU. When there are more than four sectors in scene, previous methods as well as the Cartsian/polar pillar baselines show trend of decreasing PQ and mloU as the number sectors increases. However, our PolarStream with bidirectional context padding does not show such trend: the performance remains almost the same or even better than the full-sweep method. When n = 1, PolarStream got +0.9 and +1.3 improvement in PQ and segmentation mloU compared to the Cartesian pillars baseline. When sectors become smaller and spatial context becomes limited, the improvement is more significant. When n = 32, our PolarStream with Bidirectional Context Padding outperforms all previous streaming methods by large margin, with +6.7 and +6.6 improvements in PQ and segmentation mloU. This shows that our bidiretional context padding makes better use of spatial context compared to previous methods. Our detection NDS is always the highest or at least on par with the highest among all streaming models. Interestingly, Cartesian pillars/Han et al.'s method show higher mAP than ours for 1, 4 and sectors, when the sectors have plenty of spatial view and our context padding does not have many benefits. Our PolarStream outperform all previous streaming methods detection mAP for and 32 sectors, when spatial view is limited and Bidirectional Context Padding shows more advantages. In addition, the orientation and velocity error of Han et 's is on average 14.6% and 12.9% higher than ours, which will cause problems for the downstream tracking and prediction tasks. As shown in Fig our
7

---

pooied featuros from
festures from
n
B
Concat
Input
Teatures from current soctor
notis bdded
podded featume from precesling
paskked TealLTES 56.30
Figure An illustration of how previous streaming methods, Han et al. 13 and STROBE II| enlarge spatial context of current sector.
methods offers better operating points considering both accuracy and end-to-end latency. Detailed metrics including velocity error and per-class metries are shown in Supplementary.
5.3 Discussions on Streaming
Full Sweep vs. Streaming Contrary to the findings of Han et al [3], we saw improved detection models trained on full-sweeps. We
performance using 2, 4, 8 and 16 sectors as compared hypothesize this improvement to less variation in point coordinates within a sector since all sectors are first transformed to a canonical coordinate frame before processing. This suggests that simulating streaming lidars can also serve as augmentation technique for full-sweep detection.
Diagnosis of Previous Streaming Methods We first analyze STROBE's low performance in Tab. I While all other methods aggregate points from past 10 sweeps after ego-mot mootooonon compensation (Point Warp). STROBE processes the points one sweep at a time, and aggregates information from the past sweeps by first transforming the features based on ego-motion (Feature Warp) and then fusing them with the current frame. As shown in the full-sweep case in Tab all the metrics for STROBE are significantly lower than other methods. We thus find Feature Warp inferior to Point Warp for detection and especially for velocity estimation. The average velocity error (3VE) of STROBE is 0.607 m/s, significantly higher compared to Cartesian Pillars with Point Warp (0.358 m/s). We speculate that this high velocity error is because the feature maps in STROBE don't 't encode time information on account of processing one sweep at a time as compared the other methods which encode the time lag for each accumulated point from the past 10 sweeps. For 1, 2, 4 sectors, STROBE enjoys the lowest latency because it processes fewer points compared to other methods, also shown in Fig. [ The pillar feature encoder runs faster. But this advantage disappears for more than sectors because there are also only smaller number of points processed by all other methods.
Compared to baseline Cartesian Pillars, the method of Han et al. only start to work when more than 8 sectors per sweep. For and sectors. even hurts the accuracy especially in object detection. This can be explained in Fig. 4 For 2 and 4 sectors, the feature map is fully occupied. Adding the pooled features from preceding sectors is like adding noise current sector, resulting in worse accuracy. Starting from 8 sectors, there is an empty region in current sector so adding the pooled features from preceding sectors is like padding the empty region, and therefore enlarging the context.
How Streaming Models Enlarge Context We further hypothesize not only our method and Han et work by padding context from preceding sectors, but also STROBE works by padding. As shown in Fig. 4 for 2 and sectors when the feature map is fully occupied, fusing features from previous sweep is like densifying the features. Starting from 8 sectors, when there is an empty region, fusing features is like padding the empty region. We argue that all existing streaming methods work by padding, but in different format. STROBE and Han et al. are restricted by the shape of the sectors and require empty region as placeholder, and the padded features Our method pads along the edges of feature maps and is not constrained by the shape of the sector.
added or fused to the placeholder.
Further Thoughts about Context We argue that context has two aspects. First is its feature values, for the texture information it carries. Second is its spatial relation to the object of interest. Since convolution is conslation-invariane convolutional neural networks alone do not encode spatial relation. The spatial relation is maintained in the spatial arrangement of neurons on the feature map. The stateful-RNN in Han al. must work together with the empty region as placeholder to maintain the spatial arrangement, while stateful- -NNN alone does not encode spatial relation. On the other
8

---

Detection
Semantic Segmentation
Panoptic Segmentation
30
Segmentee miuu
55
Panoptic Quality
Detection
PMT
74
R54
43
64
39
70
Runtime (H2)
Runtime (H2)
Runtime (H2)
Figure 5: Comparison of methods, full-sweep PolarStream (PLSI) and PolarStream with Bidirectional Context Padding using four sectors (PLS4), and other methods on the nuScenes dataset. We compare detection and semantic segmentation the nuScenes benchmark, and panop- tie segmentation on the nuScenes val split. We only compare with methods that report both ac- euracy and runtime. The methods in comparison CCVC, PointPainting (PNT),, Cylinder3D 37 (C3D), PolarNet 36 PPLN) SalsaNext ( CCBGSA4T Po- larNet+CBGS(P+C) and PaP 40 (PaP). We color our methods in green and other methods in red.
CenterPoint(33]CC (O)) AF2S3Net
Table 2: Comparison with state-of-the-ar methods on nuScenes Val split.
3DD Detection
Semantic Segmentation
Method
mAP
runtime(Hz)
#parameters(MIB)
mAP
runtime(Hz)
#paramcterx(MB)
CenterPoint 35
56.4
11
149
Cylinder3D 37
76.1
215
Ours-PI-S1-heave
56.2
11
149
76.8
1449
hand, although our padding along the feature map edges seems simple, is an effective solution both add feature values and maintain spatial relation of context.
5.4 Comparison with other Full-Sweep Models
As the full-sweep 3D object detection and LiDAR semantic segmentation have longer histories compared to streaming models, there are more full-sweep methods in the literature. We also compare with these methods. We present the results of our full- sweep PolarStream model with (PLSI) and best performing PolarStream model with n 4 (PLS4), with the same backbone as in PointPillars(I6) As shown in Fig. [5] our method maintains a good balance of runtime and accuracy compared to other methods on both the nuScenes detection and semantic segmentation benchmark. We achieve even faster runtime with PI.$4 while preserving almost same accuracy as PLSS. The panoptic segmentation results on the nuScenes val split show that our methods outperform all existing methods in PQ with at least 55% less runtime.
We also adopt heavier 3D ResNet [4] backbone as in CBGS 41 and compare with the state of
the methods for 3D object detection and semantic segmentation in Tab. 2 Our PhSl-heavy is able to match/beat the state-of-the-ae models for detection (CenterPoint (3S]) and segmentation (Cylinder3D(37) on the nuScenes validation set. In this work, focus on onboard applications so we only choose the same backbone as in PointPillars [6] for streaming.
5.5 Ablation Studies
The Effeet Multi-Scale Context Padding As shown in Tab the advantage of Multi-Scale Context Padding starts to show up for 8, 16 and 32 sectors, especially in segmentation. For 32 sectors, when the spatial view the most restricted, we observe the largest gain in detection mAP and segmentation mloU. Multi-Scale Context Padding does not improve or hurt the baseline polar pillars model for and 4 sectors, because the network already sees enough spatial view. As we increase the amount of context, from trailing-edge padding to bidirectional padding, observe

---

Table Ablation Study of Multi-Scale Context Padding on nuScenes Val split. CP: context padding; CP xl: tralling-edge padding: CP x2: bidirectional padding.
Panoptic Quality (PQ)
Segmentation mloU
Method
4
16
32
2
16
Polar Pillars
68.7
68.6
67.7
66.2
63.9
60.3
73.4
73.4
72.5
70.8
69.5
67.1
Ours-w/ CP
68.6
69
68.1
66.4
63.4
73.3
73.7
72.6
71.6
70
Ours-w/ CP x2
68.8
69.6
68.8
69.2
68.3
73.5
74.2
73.4
73.8
73.1
Detection mAP
Detection NDS
Method
2
4
16
32
2
4
16
32
Polar Pillars
51,2
52.1
51.9
52.5
51.9
46.7
60
61.4
61.4
60.9
60.3
55.8
Ours-w/ CP x1
52.2
53.6
52.4
52.3
49.3
60.3
61.2
60.8
60.4
58.8
Ours-w/ CPx2
51.5
53.2
52.7
53.9
52.4
60.3
61.2
60.9
61.4
more improvements. Surprisingly, we observe that with 2, 4, 8 and 16 sectors, our PolarStream with bidirectional padding even outperforms our full-sweep baseline of polar pillars in all metrics including PQ. detection mAP, NDS and segmentation IoU. suggests that streaming models can be both faster and more accurate.
The effect of Feature Undistortion and Range Stratified Convolution& Normalization We do the ablation studies with n = 1, i.e., the full-sweep case. To show how we close the gap of detection accuracy between polar pillars and Cartesian Pillars, we also list the results of Cartesian pillars with the same architecture and input size. We find that polar pillars outperforms Cartesian pillars in semantic segmentation mIoU (73.2 vs 72.1). which is also found in prior arts because points in the same polar pillar have less disagreement in the semantic label compared to those in Cartesian pillar. However, polar pillars less accurate in object detection due to the challenges we discussed. In Tab. 4 we show either Range Stratified Convolution& Normalization or Feature Undistortion helps to improve detection accuracy based on polar pillars (by 0.9 and 0.4 mAP respectively). With both techniques combined, we improve detection mAp from 48.2 2 to 50.3, narrowing the gap compared to Cartesian pillars (50.6). We also apply both techniques to Cartesian pillars and they do not improve Cartesian pillars, showing they only address the specific challenges of polar pillars, instead of improving the performance by adding more parameters to the network. Our techniques do not add noticeable runtime (0.5ms). In addition, trained with semantic segmentation. The improvement is more significant for Cartesian pillars, but Cartesian pillars suffer from slight drop in segmentation mloU.
find detection mAP can be improved when simultaneouly
Table 4: Ablation Studies on the validation split of nuScenes.
Method
seg
det
Stratified Norm&Comv
Feature Undistortion
Runtime (ms)
Det mAP
Seg mloU
29.5
73.2
Polar Pillars
38,2
48.2
38.4
49.1
38.4
48.6
38.7
50.3
44,9
51.2
73.4
29.8
72.5
Cartesian Pillars
38.1
50.6
38.5
50.6
44.5
52.3
72.1
Conclusion
In this work we propose a streaming model for simultaneous 3D object Detection, Lidar Segmentation and Panoptic Segmentation. Polar pillars is introduced as more compact representation for lidar sectors compared to previous methods. Multi-scale context padding including trailing-edge padding and bidirectional padding proposed to enhance spatial context of the streaming model with minimal latency. Additionally we make several improvements, Feature Undistortion and Range Stratified Convolution& Normalization, to address the problem of applying convolutions on polar grid. Our model showed significant improvements over previous streaming methods with lower latency.
10

---

References
[1] Alsfasser, M., Siegemund, J., Kurian, J., Kummert, A.: Exploiting polar grid structure and object shadows for fast object detection in point clouds. In: Twelfth International Conference on Machine Vision (ICMV 2019). vol. 11433, 114330G. International Society for Optics and Photonics (2020)
[2] Berman, M., Triki, A.R., Blaschko, M.B.: The lovász-softmax loss: tractable surrogate for the optimization of the intersection-over-umm measure in neural networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4413 4421 (2018)
[3] Caesar, H., Bankiti, V., Lang, A.H., Vora, S., Liong, V.E., Xu, Q. Krishnan, A., Pan, Y., Baldan, G., Beijbom, O.: nuscenes: A multimodal dataset for autonomous driving. arXiv preprint arXiv:1903.1 1027 (2019)
[4] Casas, S., Luo, W., Urtasun, R.: Intentnet: Learning to predict intention from raw sensor data. In: Proceedings of The 2nd Conference on Robot Learning. pp. 947-956 (2018)
[5] Chen, Q.. Sun, L.. Cheung, E., Yuille, A.L.: Every view counts: Cross-view consistency in 3d object detection with Processing Systems (2020)
voxelization. Advances in Neural Information
[6] Chen, Q.. Sun, L.,, Wang, Z., Jia, K., Yuille, A.: Object as hotspots: An anchor-free 3d object detection approach via firing of hotspots. In: European Conference on Computer Vision. pp. 68-84. Springer (2020)
[7] Chen, X., Ma, H., Wan, J., Li, B., Xia, T.: Multi-view 3d object detection network for au- tonomous driving. In: CVPR (2017)
[8] Cheng, R., Razani, R., Taghavi, E., Li, E., Liu, B.: 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. arXiv preprint arXiv:2102.04530 (2021)
[9] Cortinhal, T., Tzelepis, G., Aksoy, E.E.: Salsanext: Fast, uncertainty-aware semantic segmenta- tion of lidar point clouds for autonomous driving. arXiv preprint #XXv:2003.00655 (2020)
[10] Fan, L., Xiong, X., Wang, F., Wang, N., Zhang, Z.: Rangedet: In defense of range view for lidar-based 3d object detection. arXiv preprint utXiv:2103.1000 (2021)
[11] Frossard, D., Suo, S., Casas, S., Tu, J.. Hu, R., Urtasun, R.: Strobe: Streaming object detection from lidar packets. arXiv preprint arXiv:2011 1.06425 (2020)
[12] Geiger, A., Lenz, P., Stiller, C., Urtasun, R.: Vision meets robotics: The kitti dataset. The International Journal of Robotics Research 32(11), 1231-1237 (2013)
[13] Han, W., Zhang, Z., Caine, B., Yang, B., Sprunk, C., Alsharif, O., Ngiam. J., Vasudevan, V., Shlens, J., Chen, Z.: Streaming object detection for 3-d point clouds. In: European Conference on Computer Vision. pp. 423-441 Springer (2020)
[14] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition. pp. 770-778 (2016)
[15] loffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference machine learning. pp. 448-456 PMLR (2015)
[16] Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang. J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 12697-12705 (2019)
[17] Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, Focal loss for dense object detection. In: Proceedings of the IEEE international conference computer vision. pp. 2980-2988 (2017)

---

[18] Liu, W., Anguelov, D.. Erhan, D., Szegedy, C., Reed, S., Fu, C.Y., Berg, A.C.: Ssd: Single shot multibox detector. In: European conference on computer vision. pp. 21-37. Springer (2016)
[19] Luo, W., Yang, B., Urtasun, R.: Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with single convolutional net. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (June 2018)
[20] Maturana, D., Scherer, S.: Voxnet: 3d convolutional neural network for real-time object
recognition. In: 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). pp. 922-928. IEEE (2015)
[21] Meyer, G.P., Laddha, A., Kee, E., Vallespi- Benlespi-Gnzzaez,, C., Wellington, C.K.: Lasernet: An effi- cient probabilistic object detector for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 12677-12686 (2019)
[22] Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning point sets for 3d classification and segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 652-660 (2017)
[23] Rapoport-Laave,, M., Raviv, D.: It's all around you: Range-guided cylindrical network for 3d object detection. arXiv preprint arXiv:2012.0312) (2020)
[24] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cmn: towards real-tim object detection with region proposal networks. IEEE transactions on pattern analysis and machine intelligence (9()). 1137-1149 (2016)
[25] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer-assisted intervention. pp. 234-241. Springer (2015)
[26] Simon, M., Milz, S., Amende, K., Gross, H.M.: Complex-yolo- Real-time 3d object detection on point clouds. CoRR (2018)
[27] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo. J., Zhou, Y., Chai, Y., Caine, B., al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 2446 2254 (2020)
[28] Tang, H., Liu, Z., Zhao, S., Lin, Y., Lin, J., Wang, H., Han, S.: Searching efficient 3d architec- tures with sparse point-voxel convolution. In: European Conference Computer Vision. pp. 685-702. Springer (2020)
[29] Vora, S., Lang, A.H., Helou, B., Beijbom, O.: Pointpainting: Sequential fusion for 3d object detection. In: Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition. pp. 4604-6611 (2020)
[30] Yan, Y.. Mao, Y., Li, B.: Second: Sparsely embedded convolutional detection. Sensors 18(10), 3337 (2018)
[31] Yang. B., Liang, M., Urtasun, R.: HDNET: Exploiting HD maps for 3d object detection. In: CoRL (2018)
[32] Yang, B., Luo, W., Urtasun, R.: Pixor: Real-time 3d object detection from point clouds.
In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition. pp. 7652-7660 (2018)
[33] Yang, B., Luo, W., Urtasun, R.: PIXOR: Real-time 3d object detection from point clouds. In: CVPR (2018)
[34] Ye, Y., Chen, H., Zhang, C., Hao, X., Zhang, Z.: Sarpnet: Shape attention regional proposal
network for lidar-based 3d object detection. Neurocomputing 379, 53-63 (2020)
[35] Yin, T., Zhou, X., Krähenbühl, Center-based 3d object detection and tracking. arXiv preprint arXiv:2006.1 1275 (2020)
12

---

[36] Zhang, Y., Zhou, Z., David, P., Yue. X., Xi. Z., Foroosh, H.: Polaret: An improved grid repre- sentation for online lidar point clouds semantic segmentation. arXiv preprint arXiv:2003. 14032 (2020)
[37] Zhou, H., Zhu, X., Song, X., Ma, Y., Wang, Z., Li, H., Lin, D.: Cylinder3d: An effective 3d framework for driving- scene lidar semantic segmentation. arXiv preprint arXiv:2008.01550 (2020)
[38] Zhou, Y., Sun, P., Zhang, Y., Anguelov, D., Gao, J., Ouyang, T., Guo, J., Ngiam, J., Vasudevan, V: End-to-end multi-view fusion for 3d object detection in lidar point clouds, arXiv preprint 1rXiv:1910.06528 (2019)
[39] Zhou, Y., Tuzel, O.: Voxelnet: End-to-end learning for point cloud based 3d object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. pp. 4490 4499 (2018)
[40] Zhou, Z.., Zhang, Y., Foroosh, H.: Panoptic-polarnet Proposal-free lidar point cloud panoptic segmentation. arXiv preprint arXiv:2103.1496. (2021)
[41] Zhu, B., Jiang, Z., Zhou, X., Li. Z., Yu, G.: Class-balanced grouping and sampling for point cloud 3d object detection. arXiv preprint 19X8.0900000999 (2019)
13


Reviews:
[{"review_id": "wpwh8OFMI9q", "reviewer": "Reviewer_2ez1", "summary": "This paper studies an interesting and important problem: streaming-based 3D perception. The frequency of LiDAR scans are usually much lower than RGB cameras (10 FPS vs 30 FPS), and the speed of non-streaming LiDAR detectors will be limited to 10 FPS. The proposed approach can achieve higher FPS than the LiDAR frequency thanks to the sector-by-sector processing with PolarPillars. It utilizes multi-scale context padding to deal with the border effects, and addresses the distortion artifacts in object detection via Feature Undistortion + Range Stratified Convolution&Normalization. The proposed method achieves state-of-the-art performance (among 2D projection-based methods) on streaming-based 3D semantic segmentation and object detection on the nuScenes dataset, and the results are even comparable with non-streaming methods.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper attempts to improve the efficiency of real-world 3D perception through a streaming-based pipeline. The authors propose novel techniques such as feature undistortion and range stratified convolution & normalization to address the distortion caused by polar projection, which is simple and effective. The padding strategy takes advantage of temporal information and refines the border region, improve the performance of PolarPillars under large # of sectors significantly. This work also presents a solution to jointly do semseg, panopticseg and object detection with a single network, which can also improve the efficiency of 3D perception. \n\nFor the experiment evaluation, I believe the results for PolarPillar is much better than simple projection-based methods (such as PolarSeg and PointPillars). Achieving almost no accuracy loss at #sectors = 32 is impressive. It will be better if the authors can also provide batch size = 1 latency of PolarPillars at different #sectors in a table. I also wonder whether there's some chance that we can integrate SparseConv-based feature extractors into PolarPillars in the streaming mode. Potential experiment results on detection only will be very helpful. In the future the authors can also try extending PolarPillars to other datasets such as Waymo (detection) and SemanticKITTI (semseg + panopticseg).\n\nThere's a minor concern on the end-to-end latency calculation for PolarPillars. I'm not sure whether the authors follow the standard protocol in nuScenes detection to use 10-sweep point cloud as the input. If so, did you consider the time loading these point clouds, and what is the proportion of the data loading time? \n\nOverall, I believe this work has solid contributions and good quality.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The main limitation of this work is that the proposed method is evaluated only on the nuScenes dataset (though it is hard to find other datasets that have all 3 kind of tasks in this paper). ", "review_text": "", "extra_scores": {"ethical_concerns": "N/A", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "c_UGyTxy1H5", "reviewer": "Reviewer_8rdp", "summary": "This paper proposes a new \"PolarStream\" network architecture for lidar detection task. Stream-based lidar inputs are fed into polar-coordinate voxelizer. Several new techniques are proposed in order to improve detection quality: Multi-scale context padding, Feature Undistortion, Range Stratified Convolution & Normalization.\n", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the method this paper proposes is novel, and increamental improvements are explained in detail. Therefore I am giving it an accept.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I don't think there is any \"limitations and potential negative societal impact\" in this work. This is a very general lidar/camera detection task/model. ", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2.5 hours"}, "extra_text": {}}, {"review_id": "FNfadn0nsm", "reviewer": "Reviewer_88yB", "summary": "This work proposes a method of unwarping lidar sectors obtaining in a streaming setting (i.e., without waiting for a complete sweep to complete) in the (r,theta) (polar) space, and using standard convolution on top.  A new “contextual” padding scheme using ego-motion-compensated previous scan-data is proposed. Further, two other innovations — (i) feature undistortion to undo the effect of polar warping, and (ii) stratified processing (for differing statistics of data with distance) are proposed.\n\nExperiments for detection, segmentation, and panoptic segmentation are carried out on the nuScene dataset. There is insufficient empirical evidence to establish the place of the proposed method w.r.t previous methods (see “quality” below).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Originality\nIt is difficult to place what contributions are novel aspects of this work. This is because a number of methods for using cylindrical, spherical and hybrid coordinate systems have been proposed already [1,15,36,37,38,40] – the relation and distinction w.r.t to these methods has not been discussed. \n\nThe use of unwarped sectors (in (r, theta) space) in a streaming setting + the contextual padding seem to be the key original contributions — which might have limited novelty.\n&nbsp;\n\n### Quality\nThe empirical evaluation seems to be lacking:\n\n1. The baseline methods in Table 1 have not been validated on the original datasets — so it is difficult to place the results in context. Were the baselines implemented correctly? The proposed method has only been evaluated on nuScenes — again limiting fair comparison with previous methods (e.g., with Han et al. who evaluate on Waymo open dataset).\n\n2. In the comparison on the nuScenes dataset in Fig. 5: while the method is faster it lags behind existing methods in performance. It could be that the proposed method has traded-off accuracy with latency, e.g., by using a smaller backbone.  The exact details of the backbones and number of parameters in the baseline methods report are not made available – hence, it is again difficult to place these results.\n&nbsp;\n\n### Clarity\n1. This work builds on an existing body of methods for detection and segmentation in lidar data. The text defers the discussion of these details to references. This makes it difficult to grasp the proposed method adequately. Hence, the writing can be improved by simplifying the description, making the text more self-contained. \n\n2. The experiments section can be improved by collecting the “effect of multi-scale padding” (L259) and “ablation” (section 5.5) together.\n&nbsp;\n\n### Significance\nPer Table 1, for small ‘n’ i.e., when large sectors are used, the detection quality of the proposed method is comparable, while the segmentation accuracy is marginally higher than the baselines.  Only for small sectors does the proposed contextual padding from previous scan data aids the performance. Hence, one of the key contributions of this work seems to be the contextual padding (as also evidenced by long discussion devoted to padding – L259-267, L294-309, fig. 4), which is a novel but incremental contribution. \n\nFurther, the authors themselves have implemented the baseline methods (in Table 1), without validating on the original datasets that the baselines were evaluated on (e.g., Waymo open dataset). Hence, it is difficult to place the reported results in context. It is unclear if the baseline methods were trained with appropriate hyper-parameters etc. \n\nConsequently, the significance of this work might be limited.\n&nbsp;\n\n### Clarifications / Typos:\n1. L159: “we model $w’_k$ as …”: “as” is missing?\n2. Fig. 2:  Unclear where “unfold in BEV” is given as input? To “Pillar Feature Encoder”? Please clarify.\n3. L129: “mulit-scale” → multi-scale.\n4. How is the *independent* motion of objects (not ego-motion of the lidar) in the previous/old sector compensated (or not) for context padding? Seems like it is not, as there is no description to this end. Would this not introduce inaccuracies in the point-cloud? Please clarify.\n5. How is the ego-motion determined for bi-directional context padding (L202)?\n6. Is the piller-size on L207 dependent on ‘n’? Or is it held fixed, as ‘n’ changes in the experiments?\n7. In Table 1, why is there a significant drop in performance in the detection metrics when going from 16 to 32 sectors?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.\n\n### Suggestions:\nApplications of lidar detection and segmentation could be in sensitive areas like autonomous driving, and autonomous weapons. Hence, errors must be adequately quantified and presented (for driving), and access to performant lidar processing algorithms be monitored.", "review_text": "", "extra_scores": {"ethical_concerns": "No.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "6XsWXh3uobc", "reviewer": "Reviewer_ijm6", "summary": "This work improves the accuracy and efficiency of streaming lidar processing for visual recognition by adopting polar coordinates. Streaming lidar sectors are slices of a circle, so switching to polar coordinates is more efficient because angles and ranges align with these slices. Existing methods based on cartesian coordinates must instead process the containing rectangles of the slices. Having switched to polar coordinates, this work introduces a context representation that pads sectors with following/preceding sectors, making use of their now compatible spatial coordinates in the polar representation. Polar processing is not without its drawbacks though for standard convolutional processing, so tweaks are proposed to (1) fix feature distortion and (2) deal with size changes w.r.t. range. The combination of polar processing, contextual padding, and these polar tricks or fixes is the contributed PolarStream method. Conceptual figures illustrate the polar/cartesian difference and its implications for incorporating context and its impact on planar convolutional filtering. Experiments report (1) comparison with other streaming methods showing better accuracy/latency, (2) comparison with the state-of-the-art or near it at the time of submission for all methods showing better latency but worse accuracy, and (3) ablations of the proposed tweaks on planar convolutional filtering from polar inputs that show they indeed help polar processing and are not confounded with other effects on cartesian processing. The role of context and its implementation in prior work and this work is discussed in the text and illustrated by figures in a unifying way that could inform follow-up improvements of streaming lidar processing.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Novelty\n- Streaming lidar processing is already established by [11, 13] as acknowledged.\n- This work switches to a polar coordinate system while existing work uses cartesian coordinates.\n- The polar coordinate approach then allows for a new kind of multi-scale context (called bidirectional padding by this work) and tricks to improve convolutional processing of the polar inputs (feature undistortion and range stratification).\n  - Feature undistortion maps from polar coordinates back to cartesian coordinates for convolutional filtering. It is like an inverse of other input/coordinate transformation methods, like warped convolution, but differs in that its exact transformation is learned as a fully convolutional network.\n  - Range stratification divides convolution and normalization across ranges with distinct parameters and statistics. This is not unlike domain-specific normalization (AdaBN), but in this case it is uniquely spatially-motivated by the use of polar coordinates.\n- All-in-all the proposed changes are a new refinement of streaming/sector-wise lidar processing.\n\nSignificance\n- The proposed polarstream is 3-5x faster than the most accurate methods, which is a significant reduction in latency for time-critical tasks like perception for autonomous driving. However, the most accurate methods are 5-15 points absolute better in detection (mAP) and segmentation (IoU) performance. Among methods of comparable latency, the proposed method is the more accurate by a similar margin, making it state-of-the-art among streaming methods.\n- Only methods reporting accuracy and runtime are compared against. While latency is of particular importance for this application, it would be informative to report the most accurate method (regardless of runtime) for completeness.\n- The detection and segmentation metrics deserve more weight, as the panoptic metric is not based on dataset-provided instance labels (Section 5.1). The proposed method does improve on the detection and segmentation metrics.\n\nClarity\n- The main baselines for streaming are [11,13] but both are reproduced due to lack of code. Including the performance of the reproduction vs. the original results on the datasets reported in the papers would build confidence in the reproductions.\n- Detailed discussion of prior work and failure modes (Sec. 5.3, Figure 4, ...) help identify the difference in methods and isolate the various effects of sectors and how to process them. I found this more educational than simply reporting absolute numbers to know what is better and worse.\n- Ablations check the suitability of the proposed components, feature undistortion and range stratified conv + norm, for streaming/sector methods. These both help the polar representation, but not cartesian representation, showing the effect is due to the needs of sector-based streaming processing and not some confound like parameter count.\n- The implementation section needs an optimization subsection, or a reference to the supplement, to cover training hyperparameters.\n\nRelated Work\n- The provided references and discussion are sufficiently complete and detailed.\n- To give more context on (log)-polar processing in deep learning for vision, consider mentioning Warped Convolutions by Henriques & Vedaldi ICML'17 and polar transformer nets by Esteves et al. ICLR'18.\n- Although different, existing work on hybridizing convolution and range like SurfConv by Chu et al. CVPR'18 or Range Sparse Net by Sun et al. CVPR'21 (after submission deadline) could be relevant enough for discussion.\n\nDecision\nThis work is borderline due to the large accuracy gap between non-streaming and streaming methods and the not entirely certain reproduction of the baselines. However, the proposed methods for polar processing of streaming sectors seem to (1) better respect the structure of the data as arcs/polar sweeps and (2) improve accuracy and efficiency over existing streaming methods. \n\nFor Rebuttal\n- Please comment on the accuracy of polarstream vs. the state-of-the-art for accuracy metrics. Is it striking a good balance between accuracy and latency? Figure 5 is about this, but specifically mentions that only methods with runtime are reported—is there a more accurate method that is excluded?\n- Please explain how polarstream with bidirectional padding improves in mAP with more sectors, at least in the range 2-16? This seems surprising, and diverges from results for the other tasks. (Sec. 5.3 gives a hypothesis, but can this not be measured to check?)\n- Please gauge the fidelity/quality of the reproduced baselines in the context of their originally reported results, to ensure the new results are good indicators w.r.t. existing methods.\n\nOther Comments\n- Consider plotting in alternative colors than red/green for colorblindness. Blue/orange is a substitute in some textbooks, for instance.\n- Given the focus on lidar and autonomous driving, with contributions on how to structure the data and processing for the sensor, would this work have more impact at CVPR/ECCV/ICCV or IV (intelligent vehicles)?", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The limitations are not addressed in their own section, but the discussion and comparison with existing methods identifies directions for improvement. The ablations (Sec 5.5) against cartesian vs. polar pillars are adequate for this purpose.\n\nNo particular societal impacts are raised by this paper more than by any other on autonomous driving.", "review_text": "", "extra_scores": {"ethical_concerns": "None.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}]

Author rebuttal:

Meta-review: 4 expert reviewers suggest acceptance after rebuttal. This is a good quality paper making valuable contributions for streaming lidar-based object localization and that is promised to provide an open benchmark and codebase for the community.
Decision: Accept (Poster)