PolarMix: A General Data Augmentation Technique for LiDAR Point Clouds
Authors: Aoran Xiao
Jiaxing Huang
Dayan Guan
Kaiwen Cui
Shijian Lu
Ling Shao
Venue: NeurIPS 2022 Accept (2022)
Keywords: point cloud
LiDAR
data augmentation
semantic segmentation
object detection

Abstract: LiDAR point clouds, which are usually scanned by rotating LiDAR sensors continuously, capture precise geometry of the surrounding environment and are crucial to many autonomous detection and navigation tasks. Though many 3D deep architectures have been developed, efficient collection and annotation of large amounts of point clouds remain one major challenge in the analytics and understanding of point cloud data. This paper presents PolarMix, a point cloud augmentation technique that is simple and generic but can mitigate the data constraint effectively across various perception tasks and scenarios. PolarMix enriches point cloud distributions and preserves point cloud fidelity via two cross-scan augmentation strategies that cut, edit, and mix point clouds along the scanning direction. The first is scene-level swapping which exchanges point cloud sectors of two LiDAR scans that are cut along the LiDAR scanning direction. The second is instance-level rotation and paste which crops point instances from one LiDAR scan, rotates them by multiple angles (to create multiple copies), and paste the rotated point instances into other scans. Extensive experiments show that PolarMix achieves superior performance consistently across different perception tasks and scenarios. In addition, it can work as a plug-and-play for various 3D deep architectures and also performs well for unsupervised domain adaptation. 

PolarMix: A General Data Augmentation Technique for LiDAR Point Clouds
Aoran Xiao', Jiaxing Huang', Dayan Guan', Kaiwen Cui', Shijian Lu Shao
"Schoo of Computer Science and Engineering, Nanyang Technological University "Mohamed bin Zayed University of Artificial Intelligence 'Terminus Group, China (aoran.xiao, jiaxing. huang, kaiwen. shijian.lu] J @ @ntu.edu.se dayan.guan guaa @ @bzzuai.ac.e. ling.shao ieee.org
Abstract
LiDAR point clouds, which are usually scanned by rotating LiDAR sensors contin- uously, capture precise geometry of the surrounding environment and crucial many autonomous detection and navigation tasks. Though many 3D deep architec- tures have been developed, efficient collection and annotation of large amounts of point clouds remain one major challenge in the analytics and understanding of point cloud data. This paper presents PolarMix, a point cloud augmentation technique that is simple and generic but can mitigate the data constraint effectively across dif- ferent perception tasks and scenarios. PolarMix enriches point cloud distributions and preserves point cloud fidelity via two cross-scan augmentation strategies that cut, edit, and mix point clouds along the scanning direction. The first is scene-level swapping which exchanges point cloud sectors of two LiDAR scans that are cut along the azimuth axis. The second instance-level rotation and paste which crops point instances from one LiDAR scan, rotates them by multiple angles (to create multiple copies), and paste the rotated point instances into other scans. Extensive experiments show that PolarMix achieves superior performance consistently across different perception tasks and scenarios. In addition, it can work as a plug-and-pll for various 3D deep architectures and also performs well for unsupervised domain adaptation. Code is available at https: //thtttttu../...//// com/ xiwoaoran /po armix
Introduction
In the past decade, LiDAR sensors have been increasingly employed in various perception related applications such autonomous driving. They provide accurate and robust depth sensing of the surrounding environments which is crucial for scene understanding for autonomous navigation indoors and outdoors. With the recent advance of deep neural networks (DNNs), point cloud understanding has achieved significant progress in various perception tasks such as semantic segmentation [47, 16, 41, 53, 26, 27] and object detection [23, 48, 9]. On the other hand, training reliable DNN models requires large amount of well-annotated training data, whereas collecting and annotating large amounts of point clouds is often laborious, time-consuming and has poor scalability across tasks and domains. This has become one major constraint in LiDAR point cloud analytics and understanding.
Data augmentation (DA) [37, 3]. which aims to expand the distribution of the training data by modi- fying and creating new training samples, has been widely studied for 2D images and demonstrated great potential in training robust DNN models with limited training images. However, most existing DA methods do not work well for LiDAR point clouds, very meaningful but largely neglected task. Specifically, most existing DA methods perform global augmentation such randomly scaling, flip- ping, and rotation which cannot augment local structures or model relationships across neighbouring
SShjian Lu the corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Depth
Azimuth
A Faraway
(c) A sample LiDAR scan
Nearby cars
(a) A bird's-eye-view of LiDAR scanning
(b) Zoom-in
(d) The scan in (c) after PolarMix
Figure 1: A LiDAR sensor rotates and scans environments by the azimuth in XY plane, and the captured points (as shown in bird's-eys view in (a)) bear LiDAR-speciff properties including partial visibility (i.e., only object sides facing the LiDAR sensor have points captured) and density variation along the depth as illustrated in close-up views in (b). PolarMix mixes points across LiDAR scans along the scanning direction which enriches point distribution while preserving data fidelity effectively. For a sample LiDAR scan in (c), (d) shows one of its augmentations with PolarMix where points in orange color are cropped and copies from another LiDAR scan.
point cloud scans. Recently, several studies [50, 49, 13] explore local augmentation that creates new training samples via cut and mix of 2D images. However, the local augmentation does not work well for point clouds as it does not consider the unique scanning mechanism of LiDAR sensors (e.g., via continuous 360-degree sweeping) and specific properties of the captured point data.
This work focuses on effective and efficient data augmentation for better learning from limited LiDAR point cloud data. To this end, we design PolarMix, simple yet generic DA technique that can effectively work across different perception tasks and datasets. PolarMix achieves these unique features by capturing the essential properties of LiDAR point clouds, namely, partial visibility and density variation that are closely associated with the sweeping mechanism of LiDAR sensors. Specifically, objects LiDAR scans are incomplete where only object sides facing the LiDAR sensor are scanned with points as illustrated in Fig. ((a). In addition, point density varies with point depth as illustrated in Fig. (b)) Effective data augmentation needs to cater for these LiDAR Rppecific features to ensure the fidelity and usefulness of the augmented point clouds in network training.
Inspired by the above observations, PolarMix crops, edits, and mixes points along the LiDAR scanning direction (i.e., the azimuth in the 3D polar system in Fig. I(a)) to enrich point cloud distributions while maintaining its fidelity. consists of two cross-scan augmentation approaches. The first is scene-level swapping which exchanges point cloud sectors of two circular LiDAR scans that are cut along the azimuth axis as illustrated Fig. 2(a). The second is instance- rotation and paste which cuts point cloud instances from one LiDAR scan, rotates them along the scanning direction multiple times (to create multiple copies), and pastes the rotated instances into another LiDAR scan as illustrated in Fig. 2(b). For the sample LiDAR scan in Fig. 1(c), Fig. 1 (d) shows one of its augmentation where point cloud sectors and instances are mixed in with high fidelity. Extensive evaluations show that the PolarMix augmented point clouds improve the network training consistently across various tasks and benchmarks, more details to be described experiment part.
The contribution of this paper can be summarized in three aspects. First, we introduce PolarMix, simple yet effective point cloud augmentation technique that can enrich point cloud distributions while maintaining point cloud fidelity concurrently. Second, PolarMix is generally applicable and can work for different network architectures, perception tasks (e.g., object detection, semantic segmentation, etc.), and datasets/domains with consistent performance gains. Third, PolarMix is easy to use and can be incorporated as a plug and play by most existing point cloud networks. It also works well for unsupervised domain adaptation with state-of-the- performance.

---

2
Related Works
Data augmentation for 2D images. Data augmentation has been widely studied across different 2D computer vision tasks such as image classification [15, 40], object detection [32], and semantic segmentation [4]. It plays an important role in effective and efficient deep network training since collecting and annotating training images is often laborious and time-consuming. One typical DA approach is global augmentation that aims learn certain transformation invariance in image recognition tasks [33, 13), e.g. random cropping [22, 38, 39], random scaling[38, 391, random erasing [52], color jittering [39], etc. Another typical DA approach is local augmentation which performs various mix operations to generate new training data. For example. mixup [50, generates new data via convex combinations of the input pixels/feature embeddings and the output labels. CutMix [49] pastes rectangular crops from other images instead of mixing the whole images. Recently, several studies [51, 5, 10, 13] introduce the object concept into the cut and mix operations: [51] extends mixup and cutmix into object detection; [5, 10, 13] cut instances from one image and paste them into another for training better instance segmentation networks.
Data augmentation for 3D point clouds. Data augmentation of point clouds has also attracted increasing attention in recent years. Similar to 2D computer vision tasks, one direct approach is to adopt global augmentation in 3D space such as random scaling, rotation, and translation, which can be directly incorporated for expanding 3D objects [45, 2], indoor point clouds [8], and outdoor point clouds [1, 12, 30J. Several studies also explored to augment local structures of point clouds: Point-Augment [25] introduces an aumm--mgmenttiionnn network for shape-wise transformation and point- wise displacement; PatchAugment [36] exploits data augmentation in local areas; PointWOLF [21] applies weighted transformations in local neighbourhoods for enhancing the diversity of 3D objects. However, the aforementioned works focus on object-level augmentation which are not suitable for scene-level point clouds such as those in autonomous driving.
Recently, several studies explored the idea of mixing for augmenting point clouds. For example, PointMix Up [6] interpolates 3D objects to create new samples for training. PointCutMix [24] replaces subsets of point objects with that of other objects to enrich training data. However, both work focuses on object-level augmentation only. Several studies [48, 11, 46, 29] also explore scene-level mix but they are constrained with specific vision tasks. For example, GT-Aug [48, 11] cuts instances and pastes them into other LiDAR scans for the object detection task (requiring 3D bounding boxes for object cutting); Mix3D [29] concatenates points of two scenes as an out-of-conter the specific task of semantic segmentation. As a comparison, the proposed PolarMix can perform both object-level and scene-level augmentation. More importantly, its designs are perfectly aligned with LppAR-specific data properties including partial visibility and density variation with depth which guarantees superior fidelity and effectiveness of the augmented point clouds. Furthermore, PolarMix is generic and applicable to various computer vision tasks such as semantic segmentation and object detection, more details to be discussed in the experiment part.
augmentation for
3 PolarMix
Problem statement. Let s E Nx4 and y denote LiDAR scan with N points and its labels, respectively. Each point p. in s is a 1 x 4 vector with a 3D Cartesian coordinate relative to the scanner (ri, =4) and an intensity value of returning laser beam. The goal of PolarMix is to generate new training samples (8, v) by cutting and mixing across two training samples (sA, AAA) and (s B). The generated training samples (5, g) used for network training with the original loss function.
The polar coordinates. We adopt a 3D polar coordinate system where the position of a point is defined by three numbers (0, r, Φ): 0 is the azimuth angle from x-axis to y-axis which defines the rotation scanning angle of LiDAR sweeping; r denotes the depth which is the distance of the point to the LiDAR sensor; @ is the inclination angle between the z-axis and the point vector (æ,, V/, 2t).
PolarMix for LiDAR data augmentation. We designed two point cloud augmentation approaches in PolarMix including scene-level swapping approach Sw() and an instance-level rotate-paste approach Rp() for mixing LiDAR scans sB and their labels y B.. The combination of the two augmentation approaches in PolarMix can be defined as
5 Sw(sA, s |a, B) + Rp(s4, s (C))
(1)
ÿ = Sw(y .y By 9) @ Rp(y (C,9)
3

---

Scan A
Scan A
Scan A
Swap
Paste
Select
Rotate- copy
(a) Scene-level swapping
(b) Output
(c) Instance-level rotate-pasting
Figure 2: The proposed PolarMix consists of two data augmentation designs: (a) The scene-level swapping exchanges sectors of LiDAR scans A and B that are cut with certain azimuth angles; (c) The instance-level augmentation cuts point instances from scan B, rotates them about the z-axis by multiple azimuth angles (for creating multiple copies of the cut point instances), and pastes the cut and rotated instances into scan A; The augmentations of sean by the two proposed augmentation approaches are shown in (b).
where @ denotes concatenation operation. C and Ω represent class list and angle list for instance- level rotation and paste, respectively.
The scene-level swapping Swi(s, ,B|a,, (3) aims to cut a point cloud sector from azimuth angle a to azimuth angle 3 from LiDAR scan 5 and switch it with the similarly cut point cloud sector from another LiDAR scan 5 The swapping operation can be defined as follows
Sw(sA, 181a, B) ((1 [[11MM M s
) @ (M +
(2)
Sw(yA, Bla, 3) = ((1-M) o + (M o y
where 2 denotes a binary mask indicating the azimuth range (a:, (3] that is cut out from LiDAR scan sA and @ is element wise multiplication. The scene-level swapping thus exchanges two point cloud sectors of the same size from two LiDAR scans in a bird's e-eeevvvvww as illustrated in Fig. 2 (a) since the maximum scanning area in the horizontal plane circle. Note the angles a, B should be within the horizontal field-of-view of LiDAR sensor (e.g. within 360").
The instance-level augmentation Rp(s B C,Ω), illustrated in Fig. (c), crops point instances of semantic classes C from LiDAR scan S rotates them about z-axis by multiple azimuth angles Ω = (wi...., W create multiple copies, and pastes all cropped and rotated point instances into another scan A This augmentation operation can be defined by
Rp(s4, B (2) = +
Ru (MC の xB)
wED
(10 = y +
Σ (MC (M v
WEn
where MC is binary mask indicating which semantic classes of points instances to crop from LiDAR scan B, and R, represents the rotation matrix around the z-axis for an azimuth angle w.
Using polar coordinate system to augment LiDAR points has two desirable features. First, allows point cutting, rotating, and mixing be perfectly aligned with the LiDAR scanning mechanism which greatly helps preserve LiDAR-specific data properties such as partial visibility and density variation along the depth. Second, it simplifies the augmentation process with negligible computa- tional overhead: The scene-level swapping involves point slicing and concatenating only while the

---

Algorithm PolarMix.
Input: Points and labels of two LiDAR scans: y4, { "y Class list and angle list for instance-level rotate and paste: C, S2 Azimuth range for scene-level swapping: a,
Output: A new LiDAR scan for training: {5.g).
1: s. s y # Initialization
2: if' raand) S ò1 then
Scene-level swapping
3:
Calculate azimuth 0 for points in 8, 8 0,00
4: Delete points with labels in 5, if a S 0 3
5:
Cut points with labels in s y a S 0 3
6:
Update 5, y by concatenating with cut points and labels from Scan B
7:
end if
8:
if rand() S 52 then
Instance- level rotate-pasting Copy points with labels from s" s²,y" according to C' for wi in f do
9:
11: Rotate copied points with Rus duplicate their labels
Update s, by concatenating rotated points and labels
13: end for
14:
end if
instance-level augmentation can be achieved by simple dot products followed by point concatenation. Beyond that, PolarMix works in the input space which is naturally compatible with different network architectures. Algorithm summarizes the pipeline of the proposed PolarMix.
PolarMix for unsupervised domain adaptation. The proposed PolarMix can be directly applied for unsupervised domain adaptation via self-training [54]. With LiDAR data from a labeled source domain, supervised network model can be trained and applied predict pseudo labels for LiDAR data from an unlabeled target domain. PolarMix can then cut and mix LiDAR scans between the source domain (with ground-truth labels) and the target domain (with pseudo labels). Such augmented LiDAR data mitigates the inter-domain discrepancy which facilitates unsupervised domain adaptation effectively, more details be described the ensuing experiments.
Experiments
We evaluate how PolarMix benefits deep neural network training for LiDAR point cloud understanding. In Section 4. evaluate it over the semantic segmentation task across different deep architectures and benchmarking datasets. Section 4.2, we evaluate it over object detection, mainly to examine its generalization capability across different computer vision tasks. In Section 4.3, evaluate how facilitates unsupervised domain adaptation over multiple synthetic-to domain adaptation benchmarks of LiDAR data. Finally, provide in-depth analysis of different components in PolarMix in Scction 4.4.
4.1 PolarMix helps learn better representations for semantic segmentation
We first study how PolarMix helps learn better representation for semantic segmentation. The experiments were conducted over multiple deep architectures and public datasets.
4.1.1 Experimental Settings
Dataset. We evaluate PolarMix over three LiDAR datasets of driving scenes that have been widely adopted for benchmarking semantic segmentation. The first is SemanticKITTI [1] which is large-scale dataset collected in a city of Germany. It has 43, 43,551 555 LiDAR scans with 64 beams with
point-wise annotations 19 semantic classes. We follow the widely- split and use sequences 00-07, 09-10 as the training set and sequence 08 for validation. The second is nuScenes-lidarseg [12] dataset which has 40,000 scans captured 1000 scenes of 20s duration. collected with a 32 beams LiDAR sensor at 20Hz frequency with point-wise annotations of 16 semantic classes. We follow the officially split of training data and validation data. The third SemanticPOSS [30] which consists of 2,988 annotated point cloud scans of 14 semantic classes. We follow the official benchmark setting,
5

---

Table 1: Semantic segmentation over the validation set of the dataset SemanticKITTI. The baseline with either MinkNet or SPVCNN does not involve any data augmentation. CGA means conventional global augmentation which includes random scaling and random rotation. The symbol f mean that the related local data augmentation is on top of CGA, e.g., +Culix means that the network training involves both CGA and CutMix. PolarMix achieves clearly the best semantic segmentation across
both deep networks.
bi.de
mL.ele
mrack
oth-v
hidd
ml.ch
TOUL
porki
sidews
oth-
build
Tenoe
veget
pole
Methods
mloU
MinkNet 171
195.9
3.7
44.9
53.2
42.1
53.7
68.9
0.0
92.8
43.0
80.0
1.8
90.5
60.0
87.4
64.5
73.3
62.1
43.7
|55.9
+CGA
96.3
8.7
52.3
63.2
51.6
63.5
74,4
0.1
93.3
46.6
80.4
0.8
90.3
60.0
88.0
65.1
74.5
62.8
46.8
(8.99+3.0)
*CuMiix [49]
96.0
10.2
59.3
78.7
52.1
63.4
79.4
0.0
93.5
47.8
80.7
1.6
90.3
61.0
87.5
66.2
73.3
64.0
46.8
60.6(-5.7)
+Copy Paste [13]
96.6
18.4
62.8
76.3
64.6
68.9
8288
1.0
93.1
45.3
80.2
1.4
90.5
60.7
88.1
67.8
74.6
63.7
49.1
62.4(+6.5)
+Mix3DD [29]
96.3
29.6
61.8
68.5
55.4
72.7
77.7
1.0
94.3
52.9
81.7
0.9
89.1
55.5
88.3
69.3
74.6
65.2
50.3
62.4(+6.5)
+PolarMix (ours)
96.3
51.2
75.6
63.4
63.9
71.9
85.6
4,9
93.6
45.8
81.4
1.4
91.0
62.8
88.4
68.5
75.0
64.6
49.9
65.0(+9.1)
SPVCNN [-1]]
19499
9.1
55.8
66.5
33.7
61.8
75.9
0.2
93.1
45.3
79.6
0.4
91.4
62.7
87.5
66.2
72.9
62.8
42.7 55800
+CGA
96.1 21.8
57.8
69.2
49.8
66.7
80.8
0.0
93.4
44.8
80.1
0.2
90.9
62.9
88.5
64.8
75.7
63.6
46.2
60.7(+2.77
+CutMix ! [49]
96.1 21.4
59.6
71.2
54.2
66.8
81.8
0.0
93.5
49.6
81.1
2.2
90.9
63.1
87.9
66.9
74.1
63.8
49.8
61.7(+3.7)
+CopyPaste" [13]
96.0
32.4
66.4
67.1
52.9
74.8
84.3
3.6
93.3
46.9
80.2
2.5
91.1
64.1
88.1
67.0
73.9
64.0
51.6
63.2(+5.2)
MMx33D [29]
96.5
65.0
66.6
75.3
83.3
49.0
81.1
1.4
90.6
60.0
70.2
76.4
64.8
(3.7(+5.7)
+PolarMix (ours)
96.5
53,9
79.7
68.5
64,9
75.6
87.8
7.5
93.5
47.3
81.2
1.1
91.2
63.8
88.2
68.2
74.2
64.5
49.4
66.2(48.5)
i.e. sequence 03 for validation and the rest for training. For all semantic segmentation experiments, we adopt mean intersection cereseccioo-ove---eee over-union (mloU) as the evaluation metric.
Architectures and implementation details. We evaluate PolarMix over four widely adopted semantic segmentation networks: 1) MinkNet [7] which is typical voxel-based sparse CNN; 2) SPVCNN [41] which is a hybrid network with a sparse convolutional and a point-based sub-network; 3) RandLA-Net [16] which is a standard point-based network; and 4) Cylinder3D [53] which state- of-the-art cylindrical and asymmetrical 3D CNN. We adopt the default training hyper-parameters in the open-source repositories for all four networks, and the only modification is the batch size for SPVCNN and MinkNet (we change it to 8). We conducted experiments with a single Tesla 2080Ti GPU for MinkNet and SPVCNN and Tesla V100 GPU for RandLA-Net and Cylinder3D. Note training RandLA-Net and Cylinder3D takes relatively longer time, we therefore uniformly sub-sampled the same 10% of SemanticKITTI for faster experiments with these two networks.
For augmentation with scene-level swapping, we randomly crop 180 sectors from 360 for [a, ( for point swapping. For augmentation with instance-level rotate and paste, we take three rotation angles for dataset SemanticKITTI (0,, and another two rotation angles randomly picked from (0', 120%] and (120", 240")), and two rotation angles for datasets SemanticPOSS and nuScenes-lidarseg (0' and another rotation angle randomly chosen from either +90" or -90"). We set 6,, ó2 as 0.5, 1, respectively. We also examine hyper-parameter of PolarMix with details provided in the appendix.
4.1.2 Results
PolarMix improves semantic segmentation by large margins. Since data augmentation for LiDAR
semantic segmentation is a relatively under-explored task with few existing works, we selected the highly-related mixing-basee methods including Cut-Mix [49] and Copy-paste [13] in 2D vision and the pioneering work Mix3D [29] in 3D vision as baseline augmentation methods, and compared the proposed PolarMix with them. Tables and show experimental results across two networks MinkNet and SPVCNN and three datasets SemanticKITTI, auScenes--idarsee, and SemanticPOSS. It can be observed that the baseline with either MinkNet or SPVCNN (without involving any data augmentation in network training) produces fair semantic segmentation for all three datasets. However, including global augmentation with random scaling and rotation (i.e., +CGA in the two tables) improves semantic segmentation consistently across the two networks and the three datasets. On top of the global augmentation, further including local augmentation (i.e., +CutMxx + CCoyyaste"' and
*MinkNet and SPVCNN:
BeandLA--Net httpnn//ittte///gttt.ao./ooattta http://////cttuubccccnnttt-teeee "CylinderD:: http::///gttabb.oooo////oo/////s

---

Table Semantic segmentation over the validation of the datasets nuScenes- lidcares-lidarseg and Seman- ticPOSS. The baseline with either MinkNet or SPVCNN does not involve any data augmentation. CGA means conventional global augmentation which includes random scaling and random rotation. The symbol mean that the related local data augmentation is on top of CGA, e.g.. +CuMix means that the network training involves both CGA and CutMix. PolarMix achieves clearly the best semantic segmentation across both deep networks.
DA
MinkNet (7]
SPVCNN [41]
methods
nuScenes-lidarsey
SemanticPOSS
nuScenes-lidarseg
SemanticPOSS
None
67.1
52.1
68.4
50.7
+CGA
70.2(+3.1)
55.1(+3.0)
69.1(+0.7)
55.3(+4.6)
+CutMix [49]
70.4(+3.3)
56.0(+3.9)
71.7(+3.3)
54.7(+4.0)
+Copy-Pastel [13]
70.8(+3.7)
55.9(+2.8)
71.3(+2.9)
56.2(+5.5)
+Mix3D [29]
70.1(+3.0)
55.3(+3.2)
70.5(+2.1)
54.4(+3.7)
+PolarMix (Ours)
72.0(+4.9)
57.4(45.3)
72.1(43.7)
58.6(+7.9)
65
wio PolarMix
W/o PolarMix PolarMix
PolarMix
60
60
mlou(%
55
mloU(%)
55
50
45
45
40
10%
25%
50%
75%
100%
10%
25%
50%
75%
100%
Proportion of SemanticKITTKMinnNet
Proportion of SemanticKITTISSPCCNN
Figure 3: PolarMix helps reduce annotated training data effectively. For both MinkNet and SPVCNN, including PolarMix achieves similar segmentation accuracy by using around 75% annotated training data only, hence helps save around 25% efforts in training data collection and annotation.
+Mix3D1) further improves the semantic segmentation in most cases. As comparison, PolarMix introduces the best performance gains consistently across the two baseline networks and the three evaluated benchmarking datasets, demonstrating its great robustness and generalization capabilities across different network architectures and datasets.
PolarMix improves data-efficience The proposed PolarMix works reliably with different amounts of training data, and it also improves data efficiency by reducing training data and annotations effectively. As shown in Fig. 3, the data augmentation with PolarMix consistently helps across different proportions of training data of SemanticKITTI as well as two different segmentation networks MinkNet and SPVCNN. In addition, including PolarMix can achieve similar segmentation mloU while using 75% of SemanticKITTI in network training (as compared with training with 100% SemanticKITTI without using PolarMix) for both networks, hence saves 25% efforts in training data collection and annotations.
PolarMix works across deep architectures. The proposed PolarMix can work across different deep architectures beyond the voxel-based MinkNet and the sparse convolution network SPVCNN, We evaluate this feature by experimenting with two new deep architectures including the point- based network RandLA-Net [16] and the more recent 3D cylindrical convolutional architecture of Cylinder3D [53]. We train the two networks with default settings as in the officially released repositories. As shown in Table 3, incorporating PolarMix improves the segmentation performance consistently by large margins for both strong baselines (trained with 10% of SemanticKITTI data). This further verifies that PolarMix has superior generalization capability across different 3D deep architectures.
7

---

Table 3: Semantic segmentation results over the validation set of the SemanticKITTI dataset. We subsample the same 10% of the dataset for training, PolarMix consistently works across different 3D deep architectures.
Methods
mloU
RandLA-Net [16]
45,4
RandLA-Net [16]+PolarMix
50.5(+5.2)
Cylinder3D [53]
60.6
Cylinder3D [53] +PolarMix
62.5(+1.9)
4.2 PolarMix helps learn effective representations for object detection
Table 4: Object detection results on the validation set of nuScenes dataset. Incorporating PolarMix into the network training consistently improves the object detection across three different deep frameworks including PointPillar, Second, and CenterNet.
Methods
Car Truck Bus Trailer CV Ped Motor Bicycle
TC BarricrImAP
NDS
PointPillar [23]80.4
80.4 45.0 54.0
25.4
10,5
36.0
9.4
44.8
42.8
41.8
54.9
+PolarMix
80.9 50.1 59.2
33.7
13.6 69.3
37.0
6.4
44.7
42.0
43.76+1.9)
55.76690 55.7(+0.)
Second [48]
80.8 49.8 60.5
27.3
14.4 78.0
41.8
20.8
61.0
53.4
48..8
58.6
+PolarMix
81.3 53.6 68.3
34.4
20.0 76.5
38.2
14.7
59.6
56.8
50.30.15))0000--12)
CenterNet[9]
81.0 51.6 62.3 27.9
14.9
56.0
41.2
59.0
54.9
52.8
55.6
+PolarMix
80.4 53.4 68.8
32.5
17.5 79.1
58.3
44.1
57.7
62.2
65.4k+26) 61.1(.1.5)
Setup. The proposed PolarMix works in the input space with independence of specific tasks. We verify this property by evaluating it over object detection, another classical 3D understanding task that aims to predict 3D bounding box and label for each interested object instance. We perform experiments with dataset nuScenes [12] and three classical deep networks including PointPillar [23], Second [48], and CenterNet[9] For implementation, we adopted default training hyper-parameters and optimizer in the OpenPCDet repository and training with two Tesla 2080Ti GPUs (1 (11GB). We used random flip along the X and Y axis, random rotation, and random scaling for basic data augmentation. For a fair comparison, PolarMix directly implemented on top of the baseline with the same configurations. For evaluation metrics, adopted the widely used mean Average Precision (mAP) and nuScenes detection score (NDS).
Results. Table 4 shows experimental results. It can be observed that incorporating PolarMix improves both mAP and NDS consistently across the three tested deep networks. This experiment shows that the proposed PolarMix has superior generalization capability across different computer vision tasks, largely due to its cut-edit-mix strategy which enriches the distribution the training data in the input space without changing LiDAR-specific data properties.
4.3 PolarMix helps reduce domain gap
Unsupervised domain adaptation (UDA) is important research topic, aiming solve the noticeable performance drops of deep neural networks while training and testing across different domains, as a result of the distribution bias (domain shift). UDA has been widely studied in both 2D vi- sion [55, 56, 17, 18, 14, 19] and 3D vision [31, 20, 28, 46, 34, 35]. The proposed PolarMix can be easily extended for UDA by mixing labelled source point data and unlabeled target point data. We evaluate this nice feature by conducting experiments over two challenging synthetic-to teteetticctt-raall point cloud segmentation benchmarks including SynLiDAR SemanticKITTI and SynLiDAR - SemanticPOSS. SynLiDAR [46] is a synthetic LiDAR point cloud dataset that consists of 198k scans as collected from virtual scenes. shares 19 common point classes with the SemanticKITTI and 13 common point classes with the SemanticPOSS. In the experiments, we train networks with the labelled SyRR data (as the source data) and the unlabeled SemanticKIITI and SemanticPOSS data (as the target data), and perform evaluations over the validation set of KImaT and

---

Table 5: Experiments on unsupervised domain adaptation with SynLiDAR (as source) and Se- manticKITTI and SemanticPOSS (as target). PolarMix achieves clearly the best semantic segmenta- tion across both unsupervised domain adaptation setups.
Methods
SynLiDAR - SemanticKITTTSynLiiAAE -> SemanticPOSS
Source Only
20.4
20.1
ADDA [42]
22.8
24.9
Ent-Min [44]
25.5
25.5
Self-training [56]
26.5
27.1
PCT [46]
28.9
29.6
PolarMix(Ours)
31.0
30.4
SemanticPOSS. We follow the existing benchmarks [46] and adopt MinkNet as the segmentation model.
We adopt the self telf-nraining approach as described in section for unsupervised domain adaptation. Specifically, first train supervised model with the labelled source data and apply the supervised model to predict pseudo labels for the unlabelled target data. We then apply PolarMix to cut and mix between the labelled source data and the pseudo-labelled target data, and further train the model with all augmented point data. As shown in Table 5, incorporating PolarMix achieves state-of-the-ar mloUs for both SemanticKITTI and SemanticPOSS. The superior segmentation performance is largely attributed to the cut-and-min strategy in PolarMix which effectively mitigates the distribution discrepancy across LiDAR scans of different domains.
Ablation study
Table 6: Ablation study of PolarMix for semantic segmentation over SemanticKITTI dataset.
SPVCNN is trained on the sequence 00 and and tested on the validation set.
Methods
mloU
SPVCNN [41] (baseline)
48.9
w/ Scene-level swapping
50.8(+1.9)
Intance-level pasting (simple-pasting
50.9(+2.0)
w/ Intance-level pasting (rotate-pasting)
53.2(+4.3)
w/ PolarMix (complete)
54.8(+5.9)
We perform several ablation studies to examine the contribution of the two data augmentation components in the proposed PolarMix. In the ablation studies, we train SPVCNN with the sequence 00 of SemanticKITTI and evaluate the trained models over validation set of SemanticKITTI. We adopt the same training configurations described in Section 4. and Table 6 shows experimental results. With the conventional global augmentation including random rotation and random scaling, the trained SPVCNN model achieves mloU of 48.9%. On top of that, including the proposed scene-level swapping alone improves the mloU by 1.4%. In addition, including the basic version of the proposed instance-level cut-and-mix (i.e., without multiple rotations create multiple copies of the cropped object instances) alone improves the mloU by 2.0% while incorporating the full instance- level cut-and-mix improves the mloU by 4.3%. Finally, incorporating both augmentation components (i.e., the full PolarMix) improves the mloU by 5.9%, demonstrating the complementary property of the two approaches in point data augmentation.
4.5 Discussion
We conducted experiments to examine whether mixing more than two LiDAR scans further improves the segmentation performances. Specifically, we increased mixed LiDAR seans and benchmarking them without using PolarMix. The experiments were conducted with SPVCNN that is trained with sequence 00 of SemanticKITTI. As Table shows, mixing two seans produces clearly the best performance. We examined the mixed data and found that mixing more scans introduces more

---

#Scans
no mixing (baseline)
3
mloU
48,9
54,8
52.2
51.3
Table 7: Varying number mixed scans. 'no mixing' represents the vanilla training without augmentation of PolarMix.
hardly distinguishable objects. The experimental results are consistent with other mixing-band based augmentation works [50, 49, 6, 29].
5
Conclusion
This paper presents PolarMix which a data augmentation method for LiDAR point cloud learning. It produces new point cloud data by mixing LiDAR scans for training networks. There two approaches that are designed based on LiDAR data properties in PolarMix. Specifically, the scene- level swapping exchanges points within the same range of azimuth angles while the instance-level rotating-pasting selects points of certain classes from one LiDAR scan and rotates for multiple copies before pasting into another scan. Extensive experiments show the superiority of method in data augmentation for both semantic segmentation and object detection across a variety of deep frameworks and public datasets. We also extended PolarMix into unsupervised domain adaptation and achieved state-of-the-an performances in multiple synthetic-to-real LiDAR data segmentation benchmarks. We hope that the idea of PolarMix can encourage future research provide deeper insights on data augmentation for deep point cloud learning.
Acknowledgments and Disclosure of Funding
This study is supported under the RIE2020 Industry Alignment Fund Industry Collaboration Projects (IAF-ICP) Funding Initiative, as well as cash and in-kind contribution from Singapore Telecommunications Limited (Singtel), through Singtel Cognitive and Artificial Intelligence Lab for Enterprises (SCALE@NTU).
References
[1] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall, Semantickitti: A dataset for semantic seene understanding of lidar sequenees. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 9297-9307, 2019.
[2] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv: / /5/ 2015.
[3] Olivier Chapelle, Jason Weston, Léon Bottou, and Vladimir Vapnik. Vicinal risk minimization. Advances in neural information processing systems, 13, 2000.
[4] Liang-Cheen Chich Chen, George Papandreou, lasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions pattern analysis and machine intelligence, 40(4):834-848, 2017.
[5] Xiaozhi Chen, Huimin Ma, Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving, In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1915-1915,200
[6] Yunlu Chen, Vincent Tao Hu, Efstratios Gavves, Thomas Mensink, Pascal Mettes, Pengwan Yang, and Cees GM Snock. Pointmixup: Augmentation for point clouds. In European Conference on Computer Vision, pages 330-345. Springer, 2020.
[7] Christopher Choy, Jun Young Gwak, and Silvio Savarese. 4d spatio-temport convnets: Minkowski convolutional neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3075-3084, 2019.
[8] Angela Dai, Angel Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly- annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1928-5839, 2017.
10

---

[9] Kaiwen Duan, Song Bai, Lingxi Xie, Honggang Qi. Qingming Huang, and Qi Tian. Centernet: Keypoint triplets for object detection. In Proceedings the IEEE/CVF international conference computer vision, pages 6569-6578. 2019.
[10] Hao-Shu Fang, Jianhua Sun, Runzhong Wang, Minghao Gou, Yong-Lu Li, and Cewu Lu. Instaboost: Boosting instance segmentation via prohability map guided copy-pasting. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 182-691, 2019.
[11] Jin Fang. Xinxin Zuo, Dingfu Zhou, Shengze Jin, Sen Wang, and Liangjun Zhang. Lidar-aug: A general rendering-basee based augmentation framework for 3d object detection. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7710-4720, 2021.
[12] Whye Kit Fong, Rohit Mohan, Juana Valeria Hurtado, Lubing Zhou, Holger Caesar, Oscar Beijbom, and Abhinav Valada. Panoptic nuscenes: A large-scale benchmark for lidar panoptic segmentation and tracking. IEEE Robotics and Automation Letters, 7(2):79953882,, 2022.
[13] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian. Tsung-Yi Yi Lin, Ekin Cubuk, Quoe Le, and Barret Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2918-2928 2021.
[14] Dayan Guan, Jiaxing Huang, Aoran Xiao, and Shijian Lu. Domain adaptive video segmentation via temporal consistency regularization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 5053-8064, 2021.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-775 2016.
[16] Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 11 11108 1118-1117,,,11111111111111111111 2020.
[17] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu, Cross-view regularization for domain adaptive panoptic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 10133-10144, 2021.
[18] Jiaxing Huang, Dayan Guan, Aoran Xiao, and Shijian Lu. Rda: Robust domain adaptation via fourier adversarial attacking. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 8988-8999, 2021.
[19] Jiaxing Huang, Dayan Guan, Aoran Xiao, Shijian Lu, and Ling Shao. Category contrast for unsupervised domain adaptation in visual tasks. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 1214,124, 2022.
[20] Maximilian Jaritz, Tuan- Tuan-Hung Vu, Raoul de Charette, Emilic Wirbel, and Patrick Pérez. xmuda: Cross- modal unsupervised domain adaptation for 3d semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2260--2614 2020.
[21] Sihyeon Kim, Sanghyeok Lee, Dasol Hwang, Jacwon Lee, Scong Jac Hwang, and Hyunwoo J Kim. Point cloud augmentation with weighted local transformations. Proceedings of the IEÉE/CVF International Conference on Computer Vision, pages 548-557, 2021.
[22] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[23] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang. and Oscar Beijbom. Pointpillars: Fast encoders for object detection from point clouds. Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 12697-12705, 2019.
[24] Dogyoon Lee, Jaeha Lee, Juahyeop Lee, Hyeongmin Lee, Minhyeok Lee, Sungmin Woo, and Sangyoun Lec. Regularization strategy for point cloud via rigidly mixed samplc. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15900-15909, 2021.
[25] Ruihui Li, Xianzhi Li, Pheng-Am Heng, and Chi- Shi-Wing Fu. Pointaugment: an auto-augmentatioo framework for point cloud classification. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 6378-6387, 2020.
[26] Kangcheng Liu, Zhi Gao, Feng Lin, and Ben M Chen. Fo-conv: Large-scale lidar point clouds understand- ing leveraging feature correlation mining and geometric-aware modeling. In 2021 IEEE International Conference on Robotics and Automation (ICRA). pages 12896-12900 IEEE, 2021.
11

---

[27] Kangcheng Liu, Zhi Gao, Feng Lin, and Ben M Chen. Fg-net: A fast and accurate framework for large-scale lidar point cloud understanding. IEEE Transactions on Cyberneticx, 2022.
[28] Zhipeng Luo, Zhongang Cai, Changqing Zhou, Gongjee Zhang. Haiyu Zhao, Shuai Yi, Shijian Lu, Hongsheng Li, Shanghang Zhang. and Ziwei Liu. Unsupervised domain adaptive 3d detection with multi-level consistency. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 8866 8875, 2021.
[29] Alexey Nekrasov, Jonas Schult, Or Litany, Bastian Leibe, and Francis Engelmann. Mix3d: Out-of-context data augmentation for 3d scenes. In 2021 International Conference 3D Vision (3DV), pages 116-125. IEEE, 2021.
[30] Yancheng Pan, Biao Gao, Jilin Mei, Sibo Geng, Chengkun Li, and Huijing Zhao. Semanticposs: point cloud dataset with large quantity of dynamic instances. In 2020 IEEE Intelligent Vehicles Symposium (IV), pages 687-693. IEEE, 2020.
[31] Can Qin, Нaoxuan You, Lichen Wang. C-C Jay Kuo, and Yun Fu, Pointdan: A multi-scale 3d domain adaption network for point cloud representation. Advances in Neural Information Processing Systems, 32, 2019.
[32] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cmn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
[33] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Scan Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115(3):111-22, 2015.
[34] Cristiano Saltori, Fabio Galasso, Giuseppe Fiameni, Nicu Sebe, Elisa Ricci, and Fabio Poiesi. Cosmix: Compositional semantic mix for domain adaptation in 3d lidar segmentation. ECCV, 2022.
[35] Cristiano Saltori, Evgeny Krivoshcev, Stéphane Lathuilière, Nicu Sche, Fabio Galasso, Giuseppe Fiameni, Elisa Ricci, and Fabio Poicsi. Gipso: Geometrically informed propagation for online adaptation in 3d lidar segmentation. ECCV, 2022.
[36] Shivanand Venkanna Sheshappanavar, Vinit Veerendraveer Singh, and Chandra Kambhamettu. Patchaug- ment: Local neighhorhood augmentation in point cloud classification. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 21 21-22117, 2021.
[37] Patrice Y Simard, Yann A LeCun, John S Denker, and Bernard Victorri. Transformation invariance in pattern recognition-taagee tangent distance and tangent propagation. In Neural networks: tricks of the trade, pages 239-274. Springer, 1998.
[38] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. arXiv preprint arXiv: 1409.155666 2014.
[39] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1-9, 2015.
[40] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages 6105-61 14. PMIRR 2019.
[41] Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. European conference on computer vision, pages 685-702. Springer, 2020.
[42] Eric Tzeng, Judy Hoffmmnn, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7167-7176 2017.
[43] Vikas Verma, Alex Lamb, Christopher Beckham, Amir Najafi, loannis Mitliagkas, David Lopez-Paz. and Yoshua Bengio. Manifold mixup: Better representations by interpolating hidden states. In International Conference on Machine Learning, pages 6438-6447. PMLR, 2019.
[44] Tuan- Tuan-Hung Vu, Himalaya Jain, Maxime Bucher, Matthieu Cord, and Patrick Pérez. Advent: Adversarial entropy minimization for domain adaptation in semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2517-2526, 2019.
12

---

[45] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912-1920, 2015.
[46] Aoran Xiao, Jiaxing Huang. Dayan Guan, Fangneng Zhan, and Shijian Lu. Transfer learning from synthetic to real lidar point cloud for semantic segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 2795-2222 2803, 2022.
[47] Aoran Xiao, Xiaofei Yang, Shijian Lu, Dayan Guan, and Jiaxing Huang, Fps-net: A convolutional fusion network for surge-scale lidar point cloud segmentation. ISPRS Journal of Photogrammetry and Remote Sensing, 176:237-22 20077-249,220 2021.
[48] Yan Yan, Yuxing Mao, and Li. Second: Sparsely embedded convolutional detection. Sensors, 18(10)33337, 2018.
[49] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choc, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference computer vision, pages 6023 0032, 2019.
[50] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv: 1700009412, 2017.
[51] Zhi Zhang, Tong He, Hang Zhang, Zhongyue Zhang. Junyuan Xic, and Mu Li. Bag of freebies for training object detection neural networks. arXiv preprint arXiv: 1902.04103, 2019.
[52] Zhun Zhong, Liang Zheng, Guoliang Kang, Shaozi Li, and Yang. Random erasing data augmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 13001-13008, 2020.
[53] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IÉEE/CVF conference on computer vision and pattern recognition, pages 9939-9948. 2021.
[54] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoe Le. Rethinking pre-training and self-training. Advances in neural information processing systems, 33:3833- 3845, 2020.
[55] Yang Zou, Zhiding Yu, BVK Kumar, and Jinsong Wang. Unsupervised domain adaptation for semantic segmentation via class-balanced self-training. In Proceedings of the European conference on computer vision (ECCV), pages 289-305, 2018.
[56] Yang Zou, Zhiding Yu, Xiaofeng Liu, BVK Kumar, and Jinsong Wang. Confidence regularized self self-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5982-5991, 2019.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contribu- tions and scope? [Yes]
(b) Did you descrihe the limitations of your work? [Yes] Please refer to the parameter learning in the appendix, where claimed that the effectiveness of our work may be influenced by the parameter settings including swapping angular range and the number of copies.
(c) Did you diseuss any potential negative societal impacts of your work? [Yes] Please refer to the broader impact section
Appendix for more details
(d) Have you read the ethies review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [N/A] The code will be available soon.
(b) Did you speciffy all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
13

---

(c) Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e-g., type of GPUs, internal cluster, cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/relessnng new assets...
(a) your work existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you re us- ing/eurating? [N/A]
(e) Did discuss whether the data you using/curating contains personally identifiable informa- tion or offensive content? [N/A]
5. If you used crowdsoureing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable?
[N/A]
(h) Did you descrihe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "Sw7wN2jXODE", "reviewer": "Reviewer_cqy3", "summary": "This paper proposes an augmentation method for point clouds, especially captured in road environments. The proposed augmentation method is to crop, cut, and mix two 3D scans at both scene-level and instance-level. This concept can be extended to unsupervised domain adaptation (UDA) by fusing source domain point cloud (known labels) and target domain point cloud (unknown labels). Series of experiments demonstrate its superiority in comparison with other point cloud augmentation methods.", "questions": "1. Intuition\n\n1-1. I understand that this method effectively improves the quality of several target tasks. However, what is the fundamental intuition behind the proposed data augmentation? Why do such strategies bring performance improvement? While authors demonstrate lots of experiments that consistently increase the accuracy, I cannot expect why such an augmentation scheme is effective and novel. \n\n1-2. What kinds of wrong estimations are fixed after applying the proposed augmentation strategy, PolarMix? For instance, is the estimation of the far-away points more accurate? Or, is this method robust to the density of 3D scans? Even though I read the additional results in the supplementary material, I cannot find any deep analysis of this issue. \n\n2. Experiments\n\n2-1. What if we fuse more than 3 scans?\n\n2-2. What if there is a misalignment after data augmentation? In Fig2 of the manuscript, PolarMix crops point clouds in a certain range of azimuth. It means that the origins of the two-point clouds are aligned. I wonder why such assumption or alignment should be required for this augmentation? Why should we crop point clouds along the azimuth axis? Do you have any reason? (This is also related to Q.1-1)\n\n2-3. Lack of comparison against recent UDA methods in Table5 of the manuscript.\nIn recent days, there are papers about UDA for 3D semantic segmentation, such as xMUDA [H]. In my opinion, the limited demonstration in Table5 of the manuscript does not fully support the authors' claims about the UDA setup. Technically speaking, PolarMix can be applicable to UDA and I agree with the authors' claims. However, I cannot judge whether this augmentation scheme is much more effective for the UDA for the 3D semantic segmentation task in comparison with previous methods [H]. Accordingly, the authors overclaim their novelty in UDA for the 3D semantic segmentation due to their limited comparisons.\n\n3. Writing\n3-1. (L208) As far as my understanding, MinkUNet also consists of sparse convolutional layers as SPVCNN did. However, the sentence can mislead readers as MinkUNet processes dense voxel-based architectures in contrast to SPVCNN.\n\n3-2. I wonder why the authors did not present their qualitative results in the main manuscripts. Not just the quantitative results, qualitative results can help readers' understanding of this work. I recommend authors to revise the paper if it is accepted. \n(I found the results in the supplementary materials. However, for completeness, I still think that the qualitative results should be listed in the main manuscripts.)\n\n> If authors relieve my worries, I am willing to change my score.\n\nReference\n- [H] xMUDA: Cross-Modal Unsupervised Domain Adaptation for 3D Semantic Segmentation, CVPR 2020\n", "limitations": "1. While authors only focus on the LiDAR scans that are captured in the road environments, there are other types of point cloud datasets, such as S3DIS [I] or ScanNet [J]. In my understanding, this method is not applicable to the indoor point set datasets. If so, authors should have included this problem as their limitation. \n\nReference\n- [I] 3d semantic parsing of large-scale indoor spaces, ICCV 2016\n- [J] Scannet: Richly-annotated 3d reconstructions of indoor scenes, CVPR 2017\n", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strength\n- Simple and straightforward ways of augmenting point clouds. \n- Potential extension to UDA tasks.\n- Consistent performance improvements in various tasks.\n\n2. Weakness\n- Lack of analysis and intuition behind such a design.\nOverall, the authors present extensive experiments to demonstrate the superiority of the proposed data augmentation scheme. However, I wonder why such data augmentation is helpful for point-based recognition tasks. For instance, is PolarMix effective in making the networks more robust to point cloud density? or point cloud noise? The lack of such analysis makes me suspicious of the novelty of this work. Currently, this is the dominant worrying point of this paper.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "RfG9A7s1Y6q", "reviewer": "Reviewer_gjSm", "summary": "This paper proposed a data augmentation method named PolarMix for LiDAR point cloud perception. It includes two separate operations: a scene-level swapping that first cuts point cloud sectors of two LiDAR scans w.r.t. the point azimuth values and then switches the cut sectors to form a new sample for training; and an instance-level copy-paste which selects instance points for certain classes from one scan, rotates them along the LiDAR scanning direction and pastes the rotated points to another scan. Experimental results show that PolarMix yields improvements for both LiDAR semantic segmentation and 3D object detection.", "questions": "Regarding weakness 1, there are some concerns w.r.t. the idea proposed in this work:\n1. In L56, the authors state that PolarMix can maintain high fidelity for the augmented samples. However, the augmented scan shown in Figure 1 (d) contains lots of instances like cars that are scattering around the center-vehicle with arbitrary directions, which might not happen in real-world scenarios. Also, the pasted instances seem to be overlapped with the background classes. As instances like cars, bicycles, and pedestrians tend to appear only on the road surface, how to ensure that they are pasted to a proper position with high fidelity?\n\n2. In L109, the LiDAR points in the polar coordinate system are defined by azimuth theta, depth r, and inclination phi. Why select only the azimuth to split point cloud sectors, but not depth or inclination or even the combination of all three?\n\n3. In L207, the authors state that the proposed method can work across different LiDAR point cloud representations. Can it be applied to range view networks? How does it ensure a proper projection after concatenating or pasting new points in the current LiDAR point cloud?\n\n4. In suppL19, randomly swapping sectors of 45 degrees could downgrade the segmentation performance. Any possible explanation on why this problem happens?\n\nRegarding weakness 2, there are some concerns w.r.t. the experiment part of this work:\n1. The main results for LiDAR semantic segmentation, e.g., Table 1, are from relatively old segmentation methods MinkNet (55.9% mIoU) and SPVCNN (58.0% mIoU), which do not yield competitive performance over state-of-the-art ones, such as Cylinder3D (65.9% mIoU). Although the authors involved RandLA-Net and Cylinder3D in later analysis, i.e., Table 3, they are only tested under the 10% data setting. Therefore, the performance gains of PolarMix from the relatively lower-score methods are not representative enough to demonstrate superiority.\n\n2. The results for 3D object detection, i.e., Table 4, only include comparisons with the baselines. Other augmentation techniques, such as copy-paste, are recommended to include for a more comprehensive comparison.\n\n3. The ablation studies in Section 4.4 is not comprehensive enough to support the effectiveness of each of the components in PolarMix. Besides, the scene-level swapping seems less effective compared to the rotate-pasting.\n\nRegarding weakness 3, there are some minor problems w.r.t. the elaboration:\n1. Omega and C should be properly defined after their first appearance, i.e., Eq. (1) in L116.\n\n2. The delta_1 and delta_2 symbols in Algorithm 1 are not defined.\n\n3. The sigma_1 and sigma_2 symbols in L185 are not defined.\n\n4. In L247, why the cut-and-mix strategy in PolarMix can mitigate the domain discrepancy? Since PolarMix does not include any domain alignment components like the domain discriminator, the performance gains seem more related to the augmentation effect but not the adaptation effect.\n", "limitations": "This paper does not include a specific description of potential limitations. In L396, the author state that the effectiveness of the proposed method might be influenced by the parameter setting, which should be regarded as and included in ablation analysis. More general limitations are recommended to be discussed, such as the applicability of the proposed method for different types of LiDAR point clouds and networks.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This paper introduces a new data augmentation method for LiDAR point clouds.\n\n2. The proposed PolarMix is tested for both LiDAR semantic segmentation and 3D object detection and the results show that the proposed method can improve the baselines for both tasks.\n\nWeaknesses:\n1. This paper explains mostly the “what” and “how” but not much “why” for their specific augmentation operations.  More design insights and analysis can improve this paper a lot.\n\n2. The experimental analysis is insufficient to support all conclusions stated in this paper.\n\n3. The elaboration of this paper is not good enough. Some assumptions are made without clear logical relations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IvWJFbiKQo", "reviewer": "Reviewer_NzUE", "summary": "This paper presents a polar coordinate system-based data augmentation approach, PolarMix, for scanning LiDAR point cloud understanding. PolarMix generates augmented data by mixing two different scans with scene-swapping and instance rotate-pasting. Extensive experiments and ablation studies validate the performance gain by PolarMix for three tasks（semantic segmentation, object detection, domain adaption）on three datasets.", "questions": "Besides the detailed ablations studies in the work, I am also curious about the following questions:\n- This work tried cutting and swapping points along azimuth directions, how about further subdividing it along inclination direction.\n- Is that possible to mix more than two scans? For example,  generate a new can by mixing each quarter from 4 different scans.\n- What do σ_1 and  σ_2 indicate (L. 185)?\n- For the rotation angles of rotate-pasting, any reason about why choosing values as L.183-185 for three datasets? ", "limitations": "The authors didn’t discuss the limitations of their research and potential negative societal impacts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The paper is well-written and easy to follow.\n- The augmentation strategy is beneficial not only for semantic segmentation tasks but also for object detection and domain adaption. I like the idea of mixing two domains’ data scan-wisely to bridge the domain gap.\n- Since PolarMix is a DA strategy, generally it is model-agnostic and applicable to any model.\n\nWeaknesses:\n- The baselines for comparison of w/  and w/o PolarMix are not the latest ones. I am wondering how is the performance gain by DA of  PolarMix under some latest backbones with higher baseline performance, the improvement will still significant?\n- The contribution is a little limited, considering the relevant work. PolarMix is kind of the combination of CutMix, Copy-Paste and Mix3D from the perspective of idea. \n- At least one relevant work is not mentioned and compared for object detection task. [1]\n\n[1]J. Fang, X. Zuo, D. Zhou, S. Jin, S. Wang and L. Zhang, LiDAR-Aug: A General Rendering-based Augmentation Framework for 3D Object Detection, CVPR2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Hm4TOQrbev", "reviewer": "Reviewer_YDbX", "summary": "- This paper introduces an approach to augmenting cylindrical LiDAR point cloud to acquire boosted performance on 3D semantic segmentation and 3D detection. The proposed approach called PolarMix enables cut, edit, and mix point clouds along the scanning direction. The augmentation happens on the scene-level and instance-level so that the augmented data provides a variety of combinations of the augmented scenes. The proposed approach is superior to conventional global rotation and scale augmentation, CutMix [37], Copy-Paste [13], and Mix3D [21].", "questions": "1. Did the authors try to mix more than two scenes? The proposed Algorithm only shows how to mix two scans, but I wonder if the idea could be generalized.\n2. It is interesting to see that the rotated instances help for the data augmentation. However, I presume that the instances in a highly cluttered or distant scene would not be that effective. Such instances would make unrealistic samples because they are partially observed in a particular viewing direction. Some analysis of the data effectiveness or performance gain versus the distribution of the distant objects would be interesting.\n3. Please check the weakness section and answer the questions there too.\n", "limitations": "- The paper does not address the limitation of the proposed approach. The concern about the computational overhead is clarified and stated. If it has drawbacks in any facts, the limitation section needs to describe them.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The paper comprehensively summarizes the related work in the point cloud augmentation field. The paper is self-contained, so the readers readily follow the problem and the recent advances.\n2. The paper is straightforward to understand. I enjoyed reading the paper. The idea of mixing the 3D scenes is already known, but mixing on the polar coordinate seems to be very effective on the cylindrical LiDAR point clouds. \n3. The approach shows compelling results on the SemanticKITTI [1], nuScenes-lidarseg [2], SemanticPOSS [22], SynLidar datasets. The proposed augmentation approach is applied with recent 3D semantic segmentation networks, such as MinkNet [8], SPVCNN [30], RandLA-Net [15], Cylinder3D [41]. For the task of 3D detection, PointPillar [18], Second [36], CenterNet [10] are applied. The gain is clear, and it outperforms other baselines.\n4. The approach shows the effectiveness of the unsupervised domain adaptation as well. Since the approach can mix labeled source data and unlabeled target data, the approach can be readily applied to the various combinations of the domains.\n5. The proposed augmentation approach improves data efficiency, as demonstrated in the experiment section. With the small amount of 3D scans, the PolarMix can produce a similar performance.\n6. The paper explains the proposed idea in detail. The supportive figures (such as Figures 1 and 2) help to understand the approach better.\n\n**Weakness**\n1. The approach is a simple extension of the idea of the mix of 3D scenes and rotating bounding boxes (limited to the azimuth angles). The idea is not entirely new, and the target domain is limited to the cylindrical LiDAR datasets, not the general 3D scenes. However, the cylindrical LiDAR domain is one of the exciting domains for the task of intelligent mobility systems, and the custom design of 3D detection for the cylindrical LiDAR data also forms a research field. Therefore, I think it is not a critical weakness.\n2. It is unclear how the baseline approaches for the semantic segmentation are selected. In addition to the CutMix [37], Copy-Paste [13], Mix3D [21], there are possible options to be applied. Similarly, approaches to augment the 3D detection task, such as GT-Aug [36, 12], CutMix, or approaches of [6, 11, 13] could be applied. The paper needs some clarification on how the baseline approaches were selected.\n3. It is recommended to indicate the computational overhead when the proposed approach is applied. For instance, when training MinkNet, compared with the vanilla MinkNet training, what percentage of the total time is added to use PolarMix? Depending on the additional computation burden, the baseline approaches could be reevaluated.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "EPrBDQzp3qC", "reviewer": "Reviewer_Hdgz", "summary": "This paper presents a data augmentation method specifically designed for LiDAR point clouds. Two layers of data augmentation are introduced: scene level augmentation and instance level augmentation. The authors demonstrated the enhancement brought by this data augmentation method on various applications, and showed that the proposed data augmentation is superior than the state-of-the-art LiDAR data augmentation methods.\n", "questions": "Will the authors release code after publication?\n", "limitations": "This paper presented a neat idea and showed extensive experiments to justify its contribution. I do not see any strong limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The idea is simple, in a good way\n+ The evaluation is thorough: the authors demonstrated three application, namely semantic segmentation, object detection, domain gap reduction\n+ The proposed method outperforms the state-of-the-art LiDAR data augmentation methods\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept