Spherical Frustum Sparse Convolution Network for LiDAR Point Cloud Semantic Segmentation
Authors: Yu Zheng
Guangming Wang
Jiuming Liu
Marc Pollefeys
Hesheng Wang
Venue: NeurIPS 2024 poster (2024)
Keywords: LiDAR Point Cloud Semantic Segmentation
2D Projection
Quantized Information Loss
Primary area: machine_vision
TL;DR: We propose spherical frustum structure to avoid quantized information loss in conventional 2D spherical projection for LiDAR point cloud semantic segmentation.

Abstract: LiDAR point cloud semantic segmentation enables the robots to obtain fine-grained semantic information of the surrounding environment. Recently, many works project the point cloud onto the 2D image and adopt the 2D Convolutional Neural Networks (CNNs) or vision transformer for LiDAR point cloud semantic segmentation. However, since more than one point can be projected onto the same 2D position but only one point can be preserved, the previous 2D projection-based segmentation methods suffer from inevitable quantized information loss, which results in incomplete geometric structure, especially for small objects. To avoid quantized information loss, in this paper, we propose a novel spherical frustum structure, which preserves all points projected onto the same 2D position. Additionally, a hash-based representation is proposed for memory-efficient spherical frustum storage. Based on the spherical frustum structure, the Spherical Frustum sparse Convolution (SFC) and Frustum Farthest Point Sampling (F2PS) are proposed to convolve and sample the points stored in spherical frustums respectively. Finally, we present the Spherical Frustum sparse Convolution Network (SFCNet) to adopt 2D CNNs for LiDAR point cloud semantic segmentation without quantized information loss. Extensive experiments on the SemanticKITTI and nuScenes datasets demonstrate that our SFCNet outperforms previous 2D projection-based semantic segmentation methods based on conventional spherical projection and shows better performance on small object segmentation by preserving complete geometric structure. Codes will be available at https://github.com/IRMVLab/SFCNet.

Spherical Frustum Sparse Convolution Network for LiDAR Point Cloud Semantic Segmentation
Yu Zheng'; Guangming Wang";
Jiuming Liu', Mare Pollefeys', Hesheng Wang't
Department of Automation, Shanghai Jiao Tong University
University of Cambridge ETH Zürich
(zhengyu0730, wiwjjuiin,iiiiiiinn..mmm www.ingnanneeeeeeeeeee sanggneennnneeeeeeaaeeeeee. edu. cn gu4620cam ac. uk
marc. marc. pollefeys@in ethz. ch
Abstract
LiDAR point cloud semantic segmentation enables the robots to obtain fine-grained semantic information of the surrounding environment. Recently, many works project the point cloud onto the 2D image and adopt the 2D Convolutional Neural Networks (CNNs) or vision transformer for LiDAR point cloud semantic segmen- tation. However, since position but only one point can be preserved, the previous 2D projection-based seg- mentation methods suffer from inevitable quantized information loss, which results in incomplete geometric structure, especially for small objects. To avoid quantized information loss, in this paper, we propose a novel spherical frustum structure, which preserves all points projected onto the same 2D position. Additionally, hash-based representation proposed for memory-effciintt spherical frustum storage. Based on the spherical frustum structure, the Spherical Frustum sparse Convolution (SFC) and Frustum Farthest Point Sampling (F2PS) are proposed convolve and sample the points stored in spherical frustums respectively. Finally, we present the Spherical Frustum sparse Convolution Network (SFCNet) adopt 2D CNNs for LiDAR point cloud semantic segmentation without quantized infor- mation loss. Extensive experiments on the SemanticKITTI and nuScenes datasets demonstrate that our SFCNet outperforms previous 2D projection-based semantic segmentation methods based on conventional spherical projection and shows bet- ter performance on small object segmentation by preserving complete geometric structure. Codes will be available at https://iihub.oom/RMM...bbb
than point can be projected onto the same 2D
Introduction
Nowadays, 3D LiDAR point clouds are widely used sensor data autonomous robot systems. Many recent works focus on resolving perception [1 [2] and localization 3 5 tasks autonomous robot systems using LiDAR point clouds. Among them, semantic segmentation on the LiDAR point cloud enables the robot fine-grained understanding of the surrounding environment. In addition, the semantic segmentation results can be adopted for the reconstruction of the semantic map (6 7 &
9 [0] of the environments.
Inspired by the achievements of deep learning in image semantic segmentation, researchers focus on searching for effective approaches to transfer the achievements to the field of point cloud semantic segmentation. Most previous works convert the raw point cloud to regular grids, like 2D images [I]
12
13
14
15
16
18
19
20
21 and 3D voxels 22 23
24 25
to exploit Convolutional Neural
Eqqual Contribution. Corresponniig Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

LEGEND
3D View
LiDAR Point Cloud
3D View
Preserved
Points
Dropped
Points
Frustum View
2D View
Spherical Projection
Spherical
Frustum
Partial
Complete Information
Information
RangeViT
SFCNet (0 urs)
LEGEND
Person
Road
Segmentation Results
Difference between our spherical frustum and conventional spherical projection. In conventional spherical projection, the points projected onto the same 2D grid are dropped, which leads to quantized information loss, e.g., dropping the boundary between the person, small object, and the road, and results in incorrect prediction of the 2D projection- method RangeViT [21] for the person. In contrast, our spherical frustum preserves all points in the frustum, which eliminates quantized information loss and makes SFCNet correctly segment the person.
Figure 1:
Networks (CNNs) and transformers in the field of point cloud semantic segmentation. The CNNs and transformer can easily process the regular grids to effectively segment the point cloud. However, due to the limited resolution, more than one point can be projected onlo the same grid, and only one point is preserved, which results in quantized information loss to the regular grid-based point cloud semantic segmentation methods. The quantized information loss poses challenge for small object segmentation since most points belonging to the small objects can be dropped during the projection. A few methods [13] [17] are proposed to compensate for the quantized information loss by restoring complete semantic predictions from partial predictions. However, quantized information loss still exists in the feature aggregation.
To overcome quantized information loss during 2D projection, in this paper, a novel spherical frustum structure is proposed. Fig. [eg.[sshee shows the comparison between the conventional spherical projection [I] and spherical frustum. Through spherical frustum, all the points projected onto the same 2D grid are preserved. Therefore, spherical frustum can avoid quantized information loss during the projection and improve the segmentation of small objects. However, without specific designs, the spherical frustum is an irregular structure and can not be processed by CNNs. Using dense grids to store the spherical frustums is an intuitive method to regularize the spherical frustum. However, since the point number of the spherical frustums is different, each point set is required to be padded to the maximal point number of the spherical frustums before being stored in the dense grid, which results propose bash-based in many redundant memory costs. To avoid redundant memory occupancy, memory- efficiiet way. In the spherical frustum representation, which stores spherical frustums a memory hash-based spherical frustum representation, the neighbor relationship of spherical frustums and points is represented through the hash table, which enables the points to be simply stored in the original irregular point set.
In the mash-based representation, each point is uniquely identified by the hash key, which consists of the 2D coordinates of the corresponding spherical frustum and the point index in the spherical frustum point set. Thus, the points projected onto any specific 2D grids can be efficiently queried. Based on the hash-based representation, we propose the Spherical Frustum sparse Convolution (SFC) to exploit 2D CNNs on spherical frustums. SFC aggregates point features of nearby spherical frustums to obtain the local feature of the center point.
Moreover, previous 2D projection-baset segmentation methods downsample the projected point cloud based on stride-based 2D sampling, which is unable to uniformly sample the 3D point cloud. However, the stride-based sampling uniformly samples the spherical frustums. Therefore, we propose novel uniform point cloud sampling method, Frustum Farthest Point Sampling (F2PS).

---

F2PS firstly samples spherical frustums by stride, and then uniformly samples the point set inside each sampled spherical frustum by Farthest Point Sampling (FPS) [26]. Since the computing complexity
of sampling points in each spherical frustum is constant-level, F2PS is an efficient sampling algorithm with linear computing complexity.
In summary, our contributions are:
We propose a novel spherical frustum structure with a memory-efficient hash-based repre- sentation. Spherical frustum avoids quantized information loss of spherical projection and preserves complete geometric structure.
We integrate spherical frustum structure into 2D sparse convolution, and propose novel Spherical Frustum sparse Convolution Network (SFCNet) for LiDAR point cloud semantic segmentation.
An efficient and uniform 3D point cloud sampling named Frustum Farthest Point Sampling (F2PS) is proposed based on the spherical frustum structure.
SFCNet is evaluated on the SemanticKITTI 1 and nuScenes datasets, The experiment results show that SFCNet outperforms previous 2D projection methods and can better segment small objects.
2
Related Work
Point-Based Semantic Segmentation. A group of works [26
learn to segment point cloud based on the raw unstructured point cloud. However, learning of raw point cloud requires the neighborhood query with high computing complexity to learn effective features from the local point cloud structure. Therefore, the efficiency of these point-based methods is limited.
28
29
30
31
32 33
3D Sparse Voxel-Based Semantic Segmentation. Storing large-scale LiDAR point clouds in dense 3D voxels requires huge memory consumption. Therefore, Graham et [M proposes the 3D sparse voxel structure. Instead of dense grids, the hash table is adopted to represent the neighborhood relations of the 3D sparse grids. Based on the hash table, the convolved grids are recorded in the rule book. According to the rule book, the 3D sparse convolution is performed. Based on the sparse 3D voxel architecture, the methods of 3D sparse convolution and 3D attention mechanisms
22 23
24
35
36
25
37
proposed.
2D Projection- Based Semantic Segmentation.
The research of image semantic segmentation has gained great achievement. Thus, many works [1222,11 13 [5
39
40
41
42
43
18
19
20
21
17 project the point cloud onto the 2D plane and utilize 2D neural networks to process the projected point cloud. Spherical projection is widely used projection method first introduced by
16
SqueezeSeg [ The subsequent works [T] 12 13
14
43 20
21 effectively segment the point
cloud with the image semantic segmentation architecture including 2D CNNs and vision transformers.
Due to the limited resolution, the 2D projection based segmentation methods suffer from quantized information loss. With quantized information loss, networks can only process the incomplete geomet- ric structure and output partial semantic predictions, which results in the penalty of segmentation performance. The previous works only focus restoring complete semantic predictions from the partial predictions of 2D neural networks. RangeNet++ [13] proposes restore the complete predictions. The semantic predictions of dropped points voted by the predic- tions of their K-Nearest Neighbors (KNN). In addition to KNN-base post-process poot-proc,ssing, KPRNet [17] directly reprojects incomplete predictions to the complete point cloud and adopts point-based network KPConv (29] to refine the predictions. However, few works explore the method preserving the complete geometric structure during projection.
strategy to
In this paper, we propose the spherical frustum which avoids the quantized information loss of spherical projection. Our spherical frustum structure can not only preserve the complete geometric structure but also output the complete semantic predictions without any post-processing or point-based network refinement.
3

---

3 SFCNet
In this section, the spherical frustum and the hash-based representation will be first illustrated in Sec. Sec.B.1 Based on the hash-based spherical frustum representation, the spherical frustum sparse convolution and frustum farthest point sampling for LiDAR point cloud semantic segmentation will be introduced in Sec. 3.2 and B.3 respectively. Finally, the architecture of the Spherical Frustum sparse Convolution Network (SFCNet) illustrated in Sec. 3.4
3.1 Spherical Frustum
Conventional Spherical Projection. The LiDAR point cloud P is composed of N points. The k-th point in P is represented by its 3D coordinates 3y [at Mk; 2x]" and the input point features f& E Rin, where Cin represents the channel dimension of the features. The conventional spherical
projection first calculates the 2D spherical coordinates of each point:
} [1 arctan I&) (1 W
Uk = (1 [1- (arcsin [11(aacsiinz))) (xtcsiin(xt/// (Z*/Tk)) + fdown) 1 H
where (H, W) is the height and width of the projected image. the point. f [up + Sdown is the vertical field-of-view of the LiDAR sensor, where Sup and d are the up and down vertical field-of-views respectively. According to the computed 2D spherical coordinates, the point features { J.W are projected onto the 2D dense image. If multiple points have the same 2D coordinates, the conventional spherical projection only projects the point closest to the origin and drops the other points, which results in the quantized information loss.
V 2h + ui + is the range of
From Spherical Projection to Spherical Frustum. Since dropping the redundant points projected onto the same 2D position results in quantized information loss, we propose the spherical frustum to preserve all the points projected onto the same 2D position. Specifically, we organize these points as a point set and assign each point with the unique index my the point set. In addition, the 3D coordinates of each point { Uk. FA) k= are preserved as the 3D geometric information for the subsequent modules.
Hash-Basee Spherical Frustum Representation. The irregular spherical frustums can not be directly processed by the 2D CNNs. A natural idea to regularize the spherical frustums is putting them in dense grids. To store the point set of each spherical frustum in the dense grids, an extra grid dimension is required. The size of this dimension should be the maximal point number M of each spherical frustum point set. However, since most of the spherical frustum point numbers are much less than M, many grids are empty. To avoid saving these empty grids in memory, we propose the hash-based spherical frustum representation to regularize the spherical frustum, where the hash table replaces the dense grids to map the 2D coordinates to the corresponding spherical frustums and points. In the hash table, the index k of any point in the original point cloud can be queried using the key (uk, Uk: mz). which is the combination of the 2D spherical coordinates and the point index in the spherical frustum point set. Based on the hash-based representation, the spherical frustums are regularly stored in a memory-efficient way.
3.2 Spherical Frustum Sparse Convolution
Since multiple points are stored in a single spherical frustum, the conventional 2D convolution can not be directly performed on the spherical frustum structure. Therefore, we propose Spherical Frustum sparse Convolution (SFC), As shown in Fig. [ SFC can be seen as the sparse convolution on the virtual spherical plane of the center point. The feature of each convolved 2D position on the virtual spherical plane is filled with the feature of the nearest point in the corresponding spherical frustum.
Selecting Convolved Spherical Frustums. SFC first selects the convolved spherical frustums for each center point p. The 3D coordinates and the 2D spherical coordinates of the center point are (xr, =) and (u, v)) respectively. The conventional 2D convolution convolves the features of the grids in the convolution kernel. Similar to the conventional convolution, the spherical frustum of each 2D position the convolution kernel is selected perform the convolution. The coordinates of the 2D positions are {(u + Au,, v + Ac;) K where K is the kernel size and (Au,, Av;) are the shift inside the kernel. Then, the points inside each spherical frustum are queried through the hash table.
Mcanwhile, the features (f, M.M, of these points obtained, where M, is the number of the points

---

Selected Convolved Spherical Frustums
Selected Noarest Point in Each Spherical Frustum
Sparse Convolution
Coordinates
(u,v)
Kemel
Hash
Selecting
Selecting Points
Shift
Table
Frustums
{ (e) F
Virtual Spherical Plane
Spherical Frustum
Center Point
Frustum Point
Selected Point
Not Selected Point
3D Geometric Information
Figure Pipeline of Spherical Frustum sparse Convolution. The spherical frustums in the convolution kernel and the points in these spherical frustums are first selected through the hash table. Then, the nearest point in each spherical frustum is determined by the 3D geometric information. Finally, the sparse convolution is performed on the selected point features.
in the i-th spherical frustum. Notably, M, can be zero, which means that points projected onto the corresponding 2D position, and the spherical frustum is invalid. The invalid spherical frustums ignored in the subsequent convolution.
Selecting the Nearest Point in Each Spherical Frustum. After identifying the points in the spherical frustum, a feature should be selected from the features of the frustum point set for 2D convolution. PointNet++ (26) emphasizes that the local feature of the center point is expected to be aggregated from the 3D neighboring points. Inspired by PointNet+++, we select the feature of the nearest point to the center point each spherical frustum. Specifically, based on the 3D geometric information, the 3D coordinates { Mi point selection. Inspired by the post-processing of RangeNet++ [3] we select the distance of range
3 Mr of the frustum points are obtained for the nearest
j=1
r, - V 23 + v} + 23 rather than the Euclidean distance as the metric of the nearest point for efficient distance calculation. Therefore, the selected point index of each spherical frustum is arg minj |r,; -rl, where is the range of the center point. According to the indexes, the convolved features {f.) are obtained, where K' is the number of valid spherical frustums.
Sparse Convolution. Finally, the sparse convolution is performed as:
K'
W.f..
(2)
where W; is the convolution weight of the i-th valid 2D position, and f' is the aggregated feature. Through the proposed spherical frustum sparse convolution, we realize effective regularization and 2D convolution-basee feature aggregation for all points in the unstructured point cloud.
3.3 Frustum Farthest Point Sampling
Sampling is significant process of point cloud semantic segmentation. Through sampling, the network can aggregate the features of different scales and recognize objects of different sizes. Moreover, the sampling should uniformly sample the point cloud to avoid key information loss. methods sample the projected point cloud using stride-based 2D sampling. This sampling ignores the 3D geometric structure of the point cloud. In contrast, as shown in Fig. 3 our Frustum Farthest Point Sampling (F2PS) only samples the spherical frustums by stride, while the spherical frustum point set is sampled by farthest point sampling.
The previous 2D projection-bast
Sampling Spherical Frustums by Stride. Specifically, we split the 2D spherical plane into several non-overiapping windows of size S, x Sw, where (S,, are the strides. The spherical frustums in each window are merged as the downsampled spherical frustum. Meanwhile, the points inside merged spherical frustums are queried through the hash table. Then, the queried points are merged as the point set {pr)/_ in the downsampled spherical frustum, where I. is the point number in the downsampled spherical frustum.
5

---

Strid-Sampled Spherical Fristums
FPS Sampled Frustum Point Set
Sampled Spherical Frustums and Points
Stride
Sw)
Point Query
Hash Table
Sampling Frustums
Sampling
Points
(e) V
Virtual Spherical Plane
Spherical Fristum
3D Geometric Information
Frustum Point Sampled Point
Dmopeed Point
Figure 3: Pipeline of Frustum Farthest Point Sampling, According to the downsampling strides, the spherical frustums in each stride window are downsampled. Then, through the hash table, the points in each downsampled spherical frustum are queried. The queried points are sampled by Farthest Point Sampling (FPS) based on the geometric information. Finally, the uniformly sampled spherical frustums and point cloud are obtained.
Sampling Frustum Point Set by Farthest Point Sampling, The Farthest Point Sampling (FPS) [26] adopted uniformly sample the point in the downsampled spherical frustum. Since the point number of each downsampled spherical frustum is much smaller than the point number of the point cloud, performing FPS is not time-consuming. Specifically, the 3D coordinates of each point in {pi}/= are first acquired from the 3D geometric information for 3D distance calculation. Then, the [S, Sw)l points are iteratively sampled from the original point set. At each iteration, the distance of each non-sampled point towards the sampled point set is calculated. The point that has maximal distance is added the sampled set. Finally, the uniformly sampled spherical frustum point set is obtained.
F2PS integrates the stride-based spherical frustum sampling with the rPS-hased frustum point set sampling. Thus, F2PS can sample the original point cloud uniformly. In addition, since performing FPS on the frustum point costs O1) time, the computing complexity of F2PS O(N). Thus, F2PS is an efficient point cloud sampling algorithm.
3.4 Network Architecture
Based on the Spherical Frustum sparse Convolution (SFC) and Frustum Farthest Point Sampling (F2PS), the Spherical Frustum sparse Convolution Network (SFCNet) is constructed. SFCNet is an encoder-decoder architecture. The hash-base spherical frustum representation is first built for convolution and sampling in the subsequent modules. Then, the point features {f E RC} are extracted through the encoder of SFCNet, where C' is the channel dimension. The encoder consists of the residual convolutional blocks from ResNet [4], where the convolutions are replaced by the proposed SFC. In addition, the point cloud downsampled based on F2PS to extract the features of different scales. After each downsampling operation, an SFC layer is adopted to aggregate the neighbor features for each sampled point. Since F2PS can uniformly sample the point cloud, the information of the original point cloud can be fully gathered in the downsampled point cloud. In the decoder, the extracted features are upsampled, concatenated, and fed into the head layer to output the prediction of semantic segmentation.
Experiments
In this section, we first introduce the datasets adopted the experiments and the implementation details of the SFCNet. Then, the quantitative results of the two datasets and the qualitative results of the SemanticKITTI dataset are presented. Finally, the ablation studies and comparison with restoring based methods are conducted to validate the effectiveness of the proposed modules.

---

Table I: Quantative results of semantic segmentation on the SemanticKITTI 1 test set. Bold results are the best in each block of methods.
other-vehicke
other groundd
mlot (%)
motoreycle
motoreychis
vegetation
taffic-sigh sign
car
bicyele
trock
pesson
bicyclist
road
parking
sidewalk
building
fence
tnunk
tetrain
pole
Approach
Point Based
PointNet++
20.1
53.7
1.9
0.2
0.9
0.0
72.0
418
5.6 62.3
16.9 46.5
30.0
6.0
8.9
Randl.A
55.9
94.2
47.4
32.2
43.9
39.1
48.4
47.
944
90.5
61.8
74.0
24.5
89.7
60.4
83.8
63.6
68.6
51.0
50.7
KPConv
20
58.8
96.0
30.2
42.5
33.4
44.3
61.5
61.6
11.8
88.8
61.3
72.7
31.6
90.5
64.2
84,8
69.2
69.1
56.4
47.4
3D Voxel Based
Cylinder3D
67.8
67.6
64.0
58.6
91.4 65.1
32.3
85.4
62.6 65.6
(AF)*-S3Net
2
69.7
94.5
65.4
86.8
39.2
41.1 80.7 80.4
74.3
91.3
68.8
72.5 53.5
87.9
63.2
70.2
68.5
53.7
61.5
71.0
SphereFormer
25
74.8
97.5
70.1
70.5
59.6
67.7
79.0
80.4
75.3
91.8
69.7
78.2
41.3
93.8
72.8
86.7
75.1
72,4
66.8
72.9
2D Projection Baved
RangeNet++
13
52.2
91.4
25.7
34.4
25.7
23.0
38.3
38.8
4.8
91.8
75.2
27.8
87.4
58.6
80.5
55.1
64.6
47.9 55.9
PolarNet
16
54.3
93.8
40.3
30.1
22.9
28.5
43.2
40.2
5.6
90.8
61.7
74.4
21.7
90.0
61.3
84.0
65.5
67.8
51.8 57.5
Vque
14
55.9
92.5
38.7 36.5 29.6
33.0
45.6
46.2
20.1
91.7 63.4
74.8
26.4
89.0
59.4
82.0
58.7
65.4
49.6 58.9
SalsaNext
h
59.5
91.9
48.3
38.6
38.9
31.9
60.2
59.0
19.4
91.7
75.8
29.1
90.2
64.2
81.8
66.5
54.3 62.1
KPRNet 77
63.1
95.5
54.1
47.9
23.6
42.6
65.9
65.0
16.5 93.2
73.9
80.6
30.2
91.7
68.4
85.7
69.8
71.2
58.7 64.1
Lite-HDSeg
63.8
40.0
55.4
39.6
59.2
54.1
93.0
29.3
65.0
78.2
65.8
59.5 67.7
RangeViT
64.0
95.4
55.8
43.5
29.8
42.1
63.9
58.2
38.1
93.1
70.2
80.0
32.5
92.0
69.0
85.3
70.6
71.2
60.8 64.7
CENet 20
64.7
91.9
58.6
50.3
40.6
42.3
68.9
65.9
43,5
90.3 60.9
75.1
31,5
91.0
66.2
84.5
69.7
70.0
61.5 67.6
SFCNeet (Ours)
Table 2:
Quantative results of semantic segmentation on the nuScenes
validation set. Bold
results are the best in each block methods.
mlot (%)
construction
motorcycle
pedestrian
traflic- cone
driveable
other flat
sidewalk
manmade
vegetation
barrier
bicycle
trailer
truck
termin
Approach
bus
car
Voxel Based
(AF))-S3Net
22
62.2
60.3
12.6
82.3
80.0 20.1 62.0
59.0
49.0
42.2
67.4
94.2
68.0
64,1
68.6
82.9
82.4
Cylinde33D
76.1
76.4
40.3
91.3
93.8
51.3
78.0
78.9
64.9
84.4
96.8
71.6
76.4
75.4
SphereFoomer
25
78.4
77.7
43.8
94.5
93.1
52.4
86.9
81.2
65.4
73.4
85.3
97.0
73.4
75.4
75.0
91.0
89.2
2D Projection Based
RangeNet++
13
65.5
66.0
21.3
77.2
80.9
30.2
66.8
69.6
52.1
54.2
72.3
94.1
66.6
63.5
70.1
83.1
79.8
PolarNet [16
71.0
74.7
28.2
85.3
90.9
35.1
77.5
71.3
58.8
57.4
76.1
96.5
71.1
74.7
74.0
87.3
85.7
SalsaNext
72.2
85.9
88.4
42.2
72.4
72.2
76.5
96.0
70.8
71.2
RangeViT 21
75.2
75.5
40.7
88.3
90.1
49.3
79.3
77.2
66.3
65.2
80.0
96.4
71.4
73.8
73.8
89.9
87.2
SFCNet (Ours)
75.9
76.7
40.4
89.5
91.3
46.7
82.0
78.1
65.8
69.4
80.6
96.6
71.6
74.5
74.9
89.0
87.5
4.1 Datasets
We train and evaluate SFCNet on the SemanticKITTI 1 and nuScenes
27 datasets, which provide
point-wise semantic labels for large-scale LiDAR point clouds.
SemanticKITTI [T] dataset contains 4355 LiDAR point cloud scans captured by the 64-line Velodyne- HDLE64 LiDAR. Each scan contains nearly 120K points. These scans are split into 22 sequences. According to the official setting, we split sequences 00-07 and 09-10 as the training set, sequence 08 the validation set, and sequences 1 11-1 as the test set. Moreover, SemanticKITTI provides the point-wise semantic annotations of 19 classes for the LiDAR semantic segmentation task.
dataset consists of 34149 LiDAR point cloud scans collected in 1000 autonomous driving scenes using the 32-line Velodyne HDL-32F LiDAR. Fach sean contains nearly 40K points. We adopt the official setting to split the scans of the nuScenes datasct into the training and validation
NuScenes 27

---

Ground Truth
CENet
RangeViT
SFCNe (Ours)
bicycle
motorcycle
otbec-vebicle
person
bicyclist
motorcyclist
read
parking
sidewalk
other-ground
building
fence
vegetation
trunk
pole traffic-sign
Figure 4: Qualitive results on SemanticKITTI validation set. The first column presents the ground truths, while the following three columns show the error maps of the predictions from the three methods. Specifically, the reference from point color the semantic class in the ground truths is shown at the bottom. In addition, the false-segment Moreover, we use circles with the same color point out the same objects in the ground truth and the three error maps. Furthermore, the corresponding RGB images of each scene with the colored point cloud projected are demonstrated. We also show the corresponding zoomed RGB image view of circled objects if they are visible in the RGB images.
points are marked as red in the eттоr maps.
sets. In addition, in nuScenes dataset, 16-class point-wise semantic annotations are provided for the LiDAR semantic segmentation task.
On both datasets, the performance of LiDAR point cloud semantic segmentation is evaluated by mean intersection-ove-Unniee (mloU) 45
4.2 Implementation Details
SFCNet is implemented through PyTorch [66 framework. For spherical frustum, the height and width in the calculation of spherical coordinates
H 64, W 1800 for the SemanticKITTI 1024 for the nuScenes dataset. The channel dimensions C' of the extracted 128 and 256 respectively. The
dataset, and = 32, point features for KemanticKITTI and nuScenes datasets are strides (Sx, S..)) of the F2PS are all set as (2, 2). The multi-layer weighted cross-entropy loss and Lovász-Softmax loss 47 are adopted for network optimization. Adam (48 with the initial learning rate 0.001 is treated as the optimizer. The learning rate is delayed by 5% in every epoch. Random rotation, flipping, translation, and scaling are utilized for data augmentation on both datasets. Model training is conducted on a single NVIDIA Quadro RTX 8000, The training batch size is set as 4.
4.3 Quantative Results
We compare our SFCNet to the The-e-of-The-Art (SoTA) 2D projection-based, point-basee and 3D voxel-based segmentation methods on the SemanticKITTI and nuScenes datasets.

---

As shown in Tabs. I and SFCNet outperforms the previous SoTA 2D convolution based seg- mentation methods CENet 20 and SalsaNext [[5] on the SemanticKITTI and nuScenes datasets respectively. In addition, SFCNet also has better performance than the vision transformer-bant segmentation method RingeViT [2I] on both two datasets. SFCNet also outperforms the point-basee methods and realizes smaller performance gap between the 2D projection- based methods and the 3D voxel-based methods. As for the per-class IoU comparison, SFCNet has better loU than the other 2D projection- methods the small 3D objects, including the motorcycle, person (which is pedestrian in nuScenes), bicyclist, trunk, and pole. The performance improvement on these small objects results from the elimination of quantized information loss. Without quantized information loss, the complete geometric structure of the small 3D objects can be preserved, which enables more accurate segmentation. We also observe the slightly weaker performances road, parking, and terrain, on the SemanticKITTT dataset. However, since preserving complete points significantly improves the accuracies of the hard small objects, SFCNet has higher mean IoU than the previous 2D projection-basee methods.
wide-range classes, e.g.,
4.4 Qualitative Results
Fig. presents the qualitative comparison between our SFCNet and the 2D projection-based based seg- mentation methods CENet (0 and Range ViT [21] The comparison shows that the predictions of SFCNet have the minimal segmentation error among the three methods. Moreover, the circled objects in the three rows Fig. 4 demonstrate the accurate segmentation of SFCNet the persons, poles, and trunks respectively. This result further indicates our better segmentation performance of 3D small objects by eliminating the quantized information loss.
4.5 Ablation Study
In this section, we conduct the ablation study on the SemanticKITTI dataset to validate the effec- tiveness of the proposed modules. We adopt the baseline network using the conventional spherical projection and stride-based sampling. The results of ablation studies are shown in Tab. 3
Spherical Frustum Sparse Convolution
Table 3: Results of ablation studies on the Se-
(SFC). First, we replace spherical projection in the baseline with spherical frustum and adopt spherical frustum sparse convolution for fea- ture aggregation. After replacement, the mloU increases 4.3%, which indicates that spherical frustum structure avoid the quantized infor- mation loss, and thus prevent segmentation error from incomplete predictions.
manticKITTI validation set.
ID
Baseline
SFC
F2PS
mIoU (%)
56.2
2
60.5
3
62.9
Frustum Farthest Point Sampling (F2PS). After replacing the stride-based 2D sampling with F2PS, the mloU increases 2.4%. F2PS uniformly samples the point cloud and preserves the key information. Thus, the performance of semantic segmentation has been improved.
4.6 Comparision with Restoring-Based Methods
Based the same baseline network in Sec. 4.5 we compare our SFCNet with the methods that
Table 4: The performance comparison between the restoring- methods and SFCNet on the
compensate for the quantized information loss
SemanticKITTT validation set.
by restoring complete predictions from par- tial predictions, including the KNN-based post-
Method
||
mloU
processing [13] and KPConv refinement [17]
KNN-bused Posi-processing [3
59.7
KPCony Refinement [17]
60.1
Tab. shows that SFCNet has 3.2% mloU im- provement to the KNN-based post-processing and 2.8% mloU improvement to KPCOnv refine- ment. Compared to the restoring-based methods, SFCNet preserves the complete geometric structure for the feature aggregation rather than compensating for the information loss by post-processing or refinement, which results in higher performance of semantic segmentation.
SFCNet (Ours)
62.9

---

5 Conclusion
In this paper, we present the Spherical Frustum sparse Convolutional Network (SFCNet), 2D convolution-based LiDAR point cloud segmentation method without quantized information loss. The quantized information loss is eliminated through the novel spherical frustum structure, which preserves all the points projected onto the same 2D position. Moreover, the novel spherical frustum sparse convolution and frustum farthest point sampling are proposed for effective convolution and sampling of the points stored in the spherical frustums. Experiment results on SemanticKITTI and nuScenes datasets show the better semantic segmentation performance of SFCNet compared to the previous 2D projection-based semantic segmentation methods, especially on small objects. The results show the great potential of SFCNet for safe autonomous driving perception due to the accurate segmentation of small targets.
Limitations and future work. To implement the 2D convolution on the spherical frustum, only the nearest points in the neighbor spherical frustums are adopted in the spherical frustum sparse convolution. This design may limit the receptive field of the network and thus result in a slightly weaker performance of the wide-range classes. To maintain the performance on both the wide-range classes and small classes, the improvement direction to expand the receptive field based on our spherical frustum structure. To realize this, future work can lie in combining the vision network architecture with larger receptive field, like the vision transformer [49] or vision mamba [50], with spherical frustum structure. In addition, our work mainly focuses on the supervised and unimodal point cloud semantic segmentation. Future work can also lie in adopting the spherical frustum structure on the weakly- Moreover, applying the spherical frustum structure to more tasks on the LiDAR point cloud, like point cloud registration [53] [4] and scene flow estimation (55], is also a direction for future work.
[3I] and multi-modal [52] point cloud semantic segmentation.
6 Acknowledgments and Disclosure of F Funding
This work was supported in part by the Natural Science Foundation of China under Grant 62225309, 62073222, U21 A20480 and 62361 166632.
References
[1] Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss. and Jurgen Gall. Semantickitü: A dataset for semantic scene understanding of lidar sequences. In Proceedings of the /EEE/CVF International Conference on Computer Vision, pages 9297-9307, 2019.
[2] Zhiqi Li, Wenhai Wang. Hongyang Li, Enze Xie, Chonghao Sima, Tong Lu, Yu Qiao, and Jifeng Dai. Bevformer: Learning bird's e-ye'vvwe representation from multi-camert images via spatiotemporal trans- formers. In European conference computer vision, pages 1-18. Springer, 2022.
[3] Jiuming Liu, Guangming Wang, Chaokang Jiang, Zhe Liu, and Hesheng Wang. Translo: A window- based masked point transformer framework for large-scale lidar odometry. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1683-1691, 2023.
[4] Juuming Liu, Dong 7huo, Zhiheng Feng, Siting 7hu, Chensheng Peng. 7he Liu, and Hesheng Wang. Dvlo: Deep visual-lidar odometry with local-to-glohal feature fusion and bi-directional structure alignment. arXiv preprint arX: 18744, 2024.
[5] Guangming Wang, Yu Zheng, Yanfeng Guo, Zhe Liu, Yixiang Zhu, Wolfram Burgard, and Hesheng Wang- End-to-end 2d-3d registration hetween image and lidar point cloud for vehicle localization. arXiv preprint arXiv:2306.11344 2023.
[6] Xieyuanli Chen, Andres Milioto, Emanuele Palazzolo, Philippe Giguere, Jens Behley, and Cyrill Stachniss. Suma++: Efficient lidar-based semantic slam. In 2019 JEEE/RS International Conference on Intelligent Robots and Systems (IROS), pages 4530-45 4537. IEEE, 2019.
[7] Siting Zhu, Renjie Qin. Guangming Wang, Jiuming Liu, and Hesheng Wang. Semgauss-slam: Dense semantic gaussian splatting slam. arXiv preprint arXiv:2403.0744 2024.
[8] Siting Zhu, Guangming Wang. Hermann Blum, Jiuming Liu, Liang Song, Mare Pollefeys, and Hesheng Wang. Sni-slam: Semantic neural implicit slam, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 21167-211777 2024.
10

---

[9] Mingrui Li, Shuhong Liu, Heng Zhou, Guohao Zhu, Na Cheng, Tianchen Deng. and Hongyu Wang. Sgs-slam: Semantic gaussian splatting for neural dense slam, arXiv preprint arXiv:2402.03244 2024.
[10] Wenhua Wu, Qi Wang, Guangming Wang, Junping Wang. Tiankun Zhao, Yang Liu, Dongchao Gao, Zhe Liu, and Hesheng Wang. Emii-map: Large-scale road surface reconstruction based on explicit mesh and implicit encoding. arXiv preprint arXiv:2403. 1189, 2024.
[11] Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. Squeezeseg: Convolutional neural nets with recurrent erf for real-time road-object segmentation from 3d lidar point cloud. In 2018 IEEE International Conference on Robotics and Automation (ICRA). pages 1897-1893, IEEE, 2018
[12] Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeczesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from a lidar point cloud. In 2019 International Conference on Robotics and Automation (ICRA), pages 4376-44 4382.. IEEE, 2019.
[13] Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. RangeNet ++: Fast and Accurate LiDAR Semantic Segmentation. 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4213-4220. IEEE, November 2019.
[14] Chenfeng Xu, Bichen Wu, Zining Wang. Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-adaptive convolution for efficient point-cloud segmentation. In European Confer- ence on Computer Vision, pages 1-19. Springer, 2020.
[15] Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy, Salsanext: Fast, uncertainty-aware semantic segmentation of lidar point clouds. In Advances in Visual Computing: 15th International Symposium, ISVC 2020, San Diego CA, USA, October 5-7, 2020, Proceedings, Part 1I 15, pages 207- (07-222. Springer, 2020.
[16] Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yuc, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polaree: An improved grid representation for online lidar point clouds semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9601 -6100, 2020.
[17] Deyvid Kochanov, Fatemeh Karimi Nejadasl, and Olaf Booij. Kprnet: Improving projection- lidar semantic segmentation. arXiv preprint arXiv:2007. 12668, 2020.
[18] Yara Ali Alnaggar, Mohamed Atifi, Karim Amer, and Mohamed ElHelw. Multi projection fusion for real- time semantic segmentation of 3d lidar point clouds. In Proceedings of the IERF/CVF Winter Conference on Applications of Computer Vision, pages 1800-1809, 2021.
[19] Ryan Razani, Ran Cheng, Ehsan Taghavi, and Liu Binghing. Lite-hdseg: Lidar semantic segmentation using lite harmonic dense convolutions. In 202/ IEEE International Conference on Robotics and Automation (ICRA). pages 9550- 9556. IEEE, 2021.
[20] Hui-Xian Cheng. Xian- Han, and Guo- Stto-Qiang Xiao. Cenet: Toward concise and efficient lidar semantic segmentation for autonomous driving. In 2022 JEEE International Conference on Multimedia and Expo (ICME), pages 01-06. IEEE, 2022.
[21] Angelika Ando, Spyros Gidaris, Andrei Bursuc, Gilles Puy, Alexandre Boulch, and Renaud Marlet. Rangevit: Towards vision transformers for semantic segmentation in autonomous driving. Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
[22] Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. Proceddings of the IEFFICVF conference on computer vision and pattern recognition, pages 12547-12556, 2021.
[23] Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma. Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9939 9948, 2021.
[24] Maosheng Ye, Shuangjic Xu, Tongyi Cao, and Qifeng Chen. Drinet: A dual-repressnnattion iterative learning network for point cloud segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pages 7447-7456, 2021.
[25] Xin Lai, Yukang Chen, Fanbin Lu, Jianhui Liu, and Jiaya Jia. Spherical transformer for lidar-based 3d recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[26] Charles Ruizhongtai Qi. Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning point sets in metric space. Advances in neural information processing systems, 30, 2017.
11

---

[27] Holger Caesar. Varun Bankiti. Alex H Lang. Sourabh Vora, Venice Erin Liong. Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In Pruceedings of the IEEE/CVF conference on computer vision and puttern recognition, pages 11621-11631, 2020.
[28] Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 9621 9630, 2019.
[29] Hugues Thomas, Charles R Qi, Jean-Emmanuel Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution point clouds. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6411-6411 64200, 2019.
[30] Hengshuang Zhao, Li Jiang. Chi- Wing Fu, and Jiaya Jia. Pointweb: Enhancing local neighborhood features for point cloud processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pages 5565-5573, 2019.
[31] Qingyong Hu, Bo Yang. Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Learning semantic segmentation large-scale point clouds with random sampling. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[32] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip HS Torr, and Vladlen Koltun. Point transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pages 16259-162220 2021.
[33] Wenxuan Wu, Li Fuxin, and Qi Shan. Pointconvformer: Revenge of the point-based convolution. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 21802- 21813, 2023.
[34] Benjamin Graham, Martin Engelcke, and Laurens Van Der Maaten. semantic segmentation with submanifold sparse convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 222-2223, 2018.
[35] Chunghyun Park, Yoonwoo Jeong, Minsu Cho, and Jaesik Park. Fast point transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16949-16958, 2022.
[36] Xin Lai, Jianhui Liu, Li Jiang, Liwei Wang. Hengshuang Zhao, Shu Liu, Xiaojuan Qi, and Jiaya Jia. Stratified transformer for point cloud segmentation. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 8500-8509, 2022.
[37] Yanbo Wang, Wentao Zhao, Chuan Cao, Tianchen Deng, Jingehuan Wang, and Weidong Chen. Sfpnet: Sparse focal point network for semantic segmentation on general lidar point clouds. arXiv preprint urXiv:2407.11566 2024.
[38] Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan I. Yuille. Deeplah: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834 848, 2017.
[39] Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. arXiv preprint 16Xiv:1602.07360. 2016.
[40] Adam Paszke, Abhishek Chaurasia, Sangpil Kim, and Fagenio Culurciello. Enet: A deep neural network architecture for real-time semantic segmentation. arXiv preprint 166.21160 2016.
[41] Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arxiv: 1804.02767, 2018.
[42] Enze Xie, Wenhai Wang. Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers, Advances in Neural Information Processing Systems, 34: 10777-1090, 2021.
[43] Yiming Zhao, Lin Bai, and Xinming Huang. Fidnet: Lidar point cloud semantic segmentation with fully interpolation decoding. In 202/ IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 4453 4458. IEEE, 2021.
[44] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 770-778, 2016.
12

---

[45] Mark Everingham. SM Ali Eslami, Luc Van Gool, Christopher KI Williams, John Winn. and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111:98-136, 2015.
[46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury. Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
[47] Maxim Berman, Amal Rannen Triki, and Matthew Blaschko. surrogate for the optimization of the intersection-over over-union measure in neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4413 04214421,22 2018.
The lovász softmax loss: A tractable
[48] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF international conference computer vision, pages 10012-10022, 2021.
[50] Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang, Wenyu Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. arXiv preprint arXiv:2401.088 09411, 2024.
[51] Jiaming Liu, Yue Wu, Maoguo Gong, Qiguang Miao, Wenping Ma, and Cai Xu. Exploring dual repre- sentations in large-scale point clouds: simple weakly supervised semantic segmentation framework. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2330-, 2023.
[52] Yue Wu, Jiaming Liu, Maoguo Gong. Qiguang Miao, Wenping Ma, Cai Xu. Joint semantic segmen- tation using representations lidar point clouds and camera images. Information Fusion, 108: 102:102370, 2024.
[53] Jiuming Liu, Guangming Wang. Zhe Liu, Chaokang Jiang, Mare Pollefeys, and Hesheng Wang. Regformer: an efficient projection-awwre transformer network for large-scale point cloud registration. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8451 2023.
[54] Fan Lu, Guang Chen, Yinlong Liu, Lijun Zhang, Sanqing Qu, Shu Liu, and Rongqi Gu. Hregnet: A hicrarchical network for large-scale outdoor lidar point cloud registration. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16014-16023, 2021
[55] Jiuming Liu, Guangming Wang, Weicai Ye, Chaokang Jiang, Jinru Han, 7he Liu, Guofeng Zhang. Dalong Du, and Hesheng Wang. Difflow3d: Toward robust uncertainty-aware scene flow estimation with itcrative diffusion-based refinement. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15109-15119, 2024.
[56] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chich Chen, Bo Chen, Mingxing Tan, Weijun Wang. Yukun Zhu, Ruoming Pang. Vijay Vasudevan, et al. Searching for mobilenetv3. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314 1324, 2019.
[57] Hyeonwoo Noh, Seunghoon Hong, and Bohyung Han. Learning deconvolution network for semantic segmentation. In Proceedingy of the IEEE international conference on computer vision, pages 1520-1528, 2015.
[58] Dan A Alcantara, Vasily Volkov, Shubhabrata Sengupta, Michael Mitzenmacher, John D Owens, and Nina Amenta. Building efficient hash table the gpu. In GPU Computing Gems Jade Edition, pages 39-53. Elsevier, 2012.
13

---

Appendix
In the appendix, first introduce the detailed architecture of the Spherical Frustum sparse Con- volution Network (SFCNet) in Sec. A Then, the additional implementation details of SFCNet are
presented in Sec. B Next, the additional experimental results illustrated in Sec. C Finally, more visualization of the semantic segmentation results on the SemanticKITTI and nuScenes 27
datasets presented Sec. D
Extraction
Extraction
Extraction
Extraction
Context Block
Layer
Layer 2
Layer 3
Layer
X 3
comppiigg St SFC)
x
Booc, SFC.
Cordinationn
Freeumm Res
SFC Blrk
SPC Block
Bock
SEC Block
SEC SFC
Head Layer
Seynesaation
Upsampling
Upsampling
Upsampling
Semani::
SFC
SFC
SFC
B
Concatomation
(a) The Detailed Pipeline of SFCNet
SFC Layer
SFC Block
Downsampling SFC Block
3x3 SFC
BotelNom
Harbbwthh
3 SFC
BotthNom
Harbaath
DatehNoum
x P2PS
3x3 SeC
Harbowieh
3 SPC
BatdhWoom
Handowosh
(b) SFC Layer
(c) SEC Block
Downsampling SFC Block
Elementwase. Addition
Figure 5: The Detailed Architecture of SFCNet. (a) presents the detailed pipeline of SFCNet. In addition, (b), (c), and (d) show the detailed module structures of the SFC layer, SFC block, and downsampling SFC block respectively, where SFC means spherical frustum sparse convolution, and F2PS means the frustum farthest point sampling.
A Detailed Architecture
Fig. shows the detailed architecture of SFCNet. SFCNet, the spherical frustum structure of the input point cloud is first constructed. Then, the encoder, which consists of the context block and extraction layers to 4, is adopted for the point feature extraction. Next, in the decoder, the point features extracted in extraction layers 4 are upsampled the upsampling Spherical Frustum sparse Convolution (SFC). The upsampled features are concatenated with the features extracted in the context block and the extraction layer The concatenated features fed into the head layer decode the point features into the semantic predictions.
In addition, Fig. also shows the three basic modules in SFCNet, including the SFC layer, SFC block, and downsampling SFC block. Moreover, we present the detailed hyperparameters of SFCNet in Tab. 5
Basic Modules of SFCNet. Specifically, the SFC layer is composed the SFC, batch normalization, and the activation function. Inspired by (20), we use Hardswish [56] as the activation function. The formula of Hardswish is:
0
i
Hardswish(z)
i
(3)
x (I + 3)/6
otherwise
14

---

Table 5: The Detailed Hyperparameters of the Components and Modules in SFCNet. Component shows the names of components in SFCNet. Module Type shows the types of basic modules used in the components. Kernel Size shows the convolution kernel size of Spherical Frustum sparse Convolutions (SFCs) used in the modules. In the column of Stride, [1, 1] strides mean the SFC treats all the points as the center points, while [2. 2] strides show the strides used in Frustum Farthest Point Sampling (F2PS). The column of Upsampling Rate shows the upsampling rate used in the upsampling SFCs. Number of Modules shows the number of composed modules used in corresponding components. Width shows the output channel dimensions of the modules. addition, in the column of Width, C' means the channel dimensions of the extracted point features, which is 128 for the SemanticKITTI dataset and 256 for the nuScenes dataset. n means the number of semantic classes, which is 19 for the SemanticKITTI dataset and 16 for the nuScenes dataset.
Component
Module Type
Kemel Size Stride Upsampling Rate Number Modules
Width
Context Block
SFC' Layer
[3,3]
[1.1]
[C/2,C,C]
Extraction Layer
SFC Block
[3.3]
[I.]]
3
[C.CCC]
Downsarppling SEC Block
[3.3]
[2.2]
[C]
Extraction Layer
SFC Block
[3.3]
[1.1]
[C,C,C]
Downsampling SFC Block
[3,3]
[2.2]
[C]
Extraction Layer 3
SFC Block
[3.3]
[1.1]
5
ICC.C.CC
Downsampling SFC Block
[3,3]
[2.2]
[C]
Extraction Layer 4
SFC Block
[3,3]
[1.1]
2
[C,C]
Upsampling SFC for Extraction Layer
[3,3]
[1.1]
[2.2]
Upsampling
Upsampling SFC for Extraction Layer
[7.7]
[1.1]
[4.4]
Upsampling SPC for Extraction Layer
[15.15]
[1.1]
[C]
SFC Layer
[3.3]
01.1]
2
[2CC,]
Head Layer
Linear
[n]
The SFC block consists two SFC layers. In addition, the residual connection [44 is adopted in the SFC block to overcome network degradation.
The downsampling SFC block combines the downsampling of Frustum Farthest Point Sampling (F2PS) and the feature aggregation of the SFC block. Notably, in the downsampling SFC block, the first SFC treats the sampled points as the center points and the features of the point cloud before sampling as the aggregated features.
Moreover, after the downsampling, the 2D coordinates of each spherical frustum are divided by the stride to gain the 2D coordinates on the downsampled 2D spherical plane. Mcanwhile, each point is assigned a new point index in the downsampled spherical frustum point set according to the sampled order in F2PS.
Components in the Encoder of SFCNet. In the encoder, the context block consists of three SFC layers to extract the initial point features from the original point cloud. The subsequent four extraction layers are composed of 3, 3. 5, and 2 SFC blocks respectively. In addition, downsampling SFC block with (2, 2) strides is adopted in the last three layers to downsample the point cloud into different scales. Thus, the multi-scale point features are extracted.
Components in the Decoder of SFCNet. We implement the upsampling SFC in the decoder of SFCNet according to the deconvolution (57] used in the 2D convolutional neural networks. In the upsampling SFC, we first multiply the 2D coordinates of the spherical frustums in the corresponding layer by the upsampling rate to obtain the 2D coordinates on the original spherical plane. Then, each point in the raw point cloud is treated as the center point in SFC. The spherical frustums fall in the convolution kernel are convolved. As shown in Tab. we set the appropriate kernel size according to the upsampling rate for each upsampling SFC.
After the upsampling, the point features from different extraction layers are of the same size. Thus, the point features can be concatenated. In the head layer, two SFC layers and linear layer are adopted for the decoding of the concatenated features.
15

---

B Additional Implementation Details
Data Normalization. For the k-th point in the LiDAR point cloud P. the combination of the 3D coordinates 2y = [ Vk, =x]"". the range T& = Vzt + v + + and the intensity is treated as the input point feature f- Because of the different units of the different data categories, the input features should be normalized.
Specifically, for the SemanticKITTI 1 dataset, like RangeNet++ [33, we minus the features by the mean and divide the features by the standard deviation obtain the normalized features. The mean and stan- dard deviation are obtained from the statistics of each input data category on the SemanticKITTI dataset,
Table 6: The statistics of each input data cate- gory on SemanticKITTI dataset.
Statistics
y = range inlensity
Mean 10.88 -100 12.12
0.21
which are presented in Tab. 6
Standard Deviation 11.476.91 0.86
12.32
0.16
For the nuScenes 27 dataset, like Cylinder3D 223
batch normalization layer is applied on the input
point features to record the mean and standard deviation of the nuScenes dataset during training. During inferencing, the recorded mean and standard deviation are used to normalize the input point features.
Spherical Frustum Construction. We construct the spherical frustum structure by assigning each point with the 2D spherical coordinates (ug, Ux) and the point index my in the spherical frustum point set, where k is the index of the point in the original point cloud. The 2D spherical coordinates can be calculated through Eq. T Thus, the key process is to assign the point index my for each point based on the 2D spherical coordinates.
We implement this by sorting the 2D coordinates (uy, 0&) of the points. The points with smaller 24 and 2/2 are ranked ahead of the points with larger 4g and Uμ. Thus, the points with the same 2D coordinates are neighbors in the sorted point cloud. For each point, we count the number of points that have the same 2D coordinates and appear ahead or behind the point in the sorted point cloud separately. The number of the points appearing ahead is treated as the point index my of each point.
In addition, we assign each point an indicator & E {0. 1} according to the number of the points appearing behind. The point with zero point appearing behind assigned zero indicator. Otherwise, the point is assigned with an indicator equal to one. The indicator indicates the end of the frustum point set and is used for the subsequent spherical frustum point set visiting.
The sorting and the point number counting are implemented through the Graphics Processing Unit (GPU)-based parallel computing using Compute Unified Device Architecture (CUDA). Thus, the construction is efficient in practice.
Hash-Based Spherical Frustum Representation. After the construction of the spherical frustum structure. we build the hash-basee spherical frustum representation. Specifically, we construct the key-value pairs between the key (ux, Uk, m&) and the value k. The key-value pairs are inserted into hash table, which represents the neighbor relationship of spherical frustums and points.
In practice, we adopt an efficient GPU-based hash table [88 The GPU-based hash table requires both key and value to be an integer. The value satisfies the integer requirement. However, the key (ux, Uk, ma) in the hash-based spherical frustum representation is not an integer.
To adopt the GPU-based hash table for efficient processing, (ug, Uk, my) is transferred to an integer as Ug (W. M) + us M + mmg, where W is the width of the spherical projection, M is the maximal point number of the spherical frustum point sets. Through this process, any point represented by the coordinates (ug, 2k m&) can be efficiently queried through the GaU-d hash table.
Spherical Frustum Point Set Visiting. Both the SFC and F2PS require visiting all the points in any spherical frustums. Thus, we propose the spherical frustum point set visiting algorithm. The visiting obtains all the points in the given spherical frustum, whose 2D coordinates (u, u). by sequentially querying the points in the hash table.
Specifically, we first query the first point in the spherical frustum using the key (u, v, 0). If the key (u, v, 0) not the hash table, the spherical frustum on (u, v) invalid. Otherwise, the first point in the spherical frustum can be queried through the hash table.
16

---

Then, the points in the spherical frustum are sequentially visited. We first initialize the point index m 0 in the spherical frustum. At each step, the point index m increases by one. Through the hash table, the point with m-th index in the spherical frustum is queried using the key (u, v, m). Meanwhile, the indicator Ç of this point is obtained. & indicates whether (u, v, m + 1) refers to valid point. Thus, the visiting ends when the indicator of the current point is zero.
Detailed Implementation of Frustum Farthest Point Sampling. In F2PS, we first sample the spherical frustums by stride. Then, we sample the points in each sampled spherical frustum by Farthest Point Sampling (FPS) [26). As mentioned in Sec. 3.3. FPS is an iterative algorithm. The detailed process of the j-th iteration can be expressed by the following formula:
Sj = S farg max min dist(p,s)): PE sE8,-1
(4)
where P. is the spherical frustum point set to be sampled, S, and S are the sampled point sets in j-th and (j 1)-th iterations respectively. Notably, Su contains the point randomly sampled from P.- In addition, dist(p, s) is the distance between point p and point s in 3D space. The iteration starts at j = 1, and ends when the size of S₁ equals the number of sampling points.
Moreover, since the distances between the points in Sj-2 and the points in PA\S have been calculated before the j-th iteration, we just need to calculate the distance between each p in P. P and the point sampled in (j 1)-th iteration for the calculation of minses, dist 8), which is the minimal distance from point p to the point set - Thus, the computing complexity of FPS for P. of size is Onn)). Since the point number of each spherical frustum is (1), the computing complexity of FPS for the spherical frustum is also O(1), which ensures the efficiency of F2PS.
Loss Function. We use multi-layer weighted cross-entropy loss and Lovász-Softmax loss 47 to help the network learn the semantic information from different scales. To get the semantic predictions of extraction layers 1 to 4, we apply linear layer to decode the extracted point features of each extraction layer into the semantic predictions.
Specifically, for extraction layer 1, the linear layer is applied the extracted point features FL to gain the prediction L, For the other extraction layers, the linear layer is applied on the upsampled point features Fz. F'y, and F1 to obtain the predictions Lzg Lzg, and L, respectively.
Based on the predictions of each layer and the final predictions of SFCNet L,, the loss function is calculated as:
C Luco
Lwce Cic L.) Loo (Lx, L).
(5)
where Lune is the weighted cross-entropy loss, LLo is the Lovász-Sofiman loss, and L is the ground truth. In addition, the weights of weighted cross-entropy loss are calculated as w. = (fe ()+++)) +) where e the semantic class, fo is the frequency of class c in the dataset, and is small positive value to avoid zero division.
C Additional Experiments
C.1 Efficiency Comparison
We evaluate the efficiency of the proposed SFCNet with the previous works and our 2D projection- based baseline model on single Geforce RTX 4090Ti GPU.
We adopt the same baseline model used in Sec. 4.5 For RingeViT, we adopt the official code for efficiency evaluation. Notably, in the inference, Range splits the projected LiDAR image, inputs each image slice into the network gain the predictions, and merges the predictions gain the prediction of the entire projected LiDAR image. Thus, the inference time of RangeVi7) includes the time of all the processes. In addition, since Range VIT adopts the KPConv refinement [17] which restores the complete predictions from the partial predictions, we use the point number of the entire point cloud as the processed point number. For PointNet++ (26). we sample 45K points from the point cloud before inputting into the network as its original setting. For 3D voxel-based methods,
Cylinder3D (23) and SphereFormer (25). only the points preserved after the voxelization are counted since these points are exactly processed in the 3D sparse convolution network.
17

---

Table 7: Efficiency comparison. The inference time of a single LiDAR scan, the processed point number, and the normalized time, the inference time per thousand points, are evaluated on the SemanticKITTI validation set with a single Geforce RTX 4090Ti GPU.
Approach
Time (ms)/Pints 4
Normalized Time (ms/K)
PointNet++ 26
131,0/~ 45K
2.91
RandLA 31
212.2/~ 120K
Cylinder3D 23
67.5/~ 40K
1.69
SphereFormer
25
118.2/- 90K
1.20
RangeViT [2]
104,8/~ 120
0.87
Bascline
46.4/~ 90K
0.52
SFCNet (Ours)
59.7/~ 120K
0.49
Table 8: The quantitative results of different resolutions used in the baseline model with KNN-based post-processing 13 on the SemanticKITTI validation set.
Resolution
Preserved Points/All Points
mloU (%) f
64 x 1800
88K/120K
59.7
x 2048
97K/120K
58.9
64 x 4096
113K/120K
57.0
The results are presented in Tab. The results show that SFCNet costs 59.7 ms for single scan inference, which reaches real-time LiDAR scan processing. addition, our SFCNet also has the highest efficiency evaluated by normalized time (0.49 ms/K) compared to the previous 3D and 2D methods and the 2D baseline model, which indicates that SFCNet can adopt the 2D projection property to efficiently segment the large-scale point cloud.
C.2 Analysis on Different Resolutions of the Baseline Model
Since the limited projection resolution the reason for quantized information loss, expanding the resolution of the projected range image can preserve case the quantized information loss. However, expanding the resolution increases the sparsity of the projected points and makes the convolution hard aggregate the local features. Thus, resolution expansion is not feasible solution for resolving quantized information loss. To validate this, we expand the image horizon resolution of the baseline model to 2048 and 4096 and conduct the ablation studies of different resolutions on the SemanticKITTI validation set to show the effect of a larger resolution. As shown Tab. X the increment of resolution preserves more points but results in worse performances. In contrast, SFCNet not only overcomes quantized information loss but also effectively aggregates local features with a suitable resolution by preserving all points using spherical frustum.
points during the spherical projection and
C.3 Additional Ablation Studies
In this subsection, conduct additional ablation studies to evaluate the sensitivity of our SFCNet to the key parameters.
Stride Sizes in Frustum Farthest Point Sampling (F2PS). The ablation studies of four different settings of the stride sizes in the F2PS on the three downsampling layers are conducted, including (1,2), (2, 1), (2, 4), and (4, 2). The results are shown in Tab. 9 The results show on all the down- sampling layers, the (2, 2) stride sizes show a better segmentation performance than the other stride size settings. (2, 2) stride sizes suitably downsample the point cloud in the vertical and horizon dimensions. Higher or lower downsampling rates result in the oversampling or undersampling the point cloud respectively.
Number of Points in the Spherical Frustums. In the spherical frustum structure, the number of points in the frustum is unlimited and only depends on how many points are projected onto the corresponding 2D location. maximal number of points in each spherical frustum and the points exceeding the maximal point number are dropped. As shown Tab. TO preserving more points in the spherical frustum results in better segmentation performance, since more complete geometry information is preserved. These
analyze the effect of the number of points in the frustum, we set
18

---

Table 9: Ablation study on the stride sizes of the Frustum Farthest Point Sampling in the downsam- pling layers on the SemanticKITTI validation set.
mloU (%)
Stride Sizes (Sh+ Sw)
Layer
Layer
Layer 3
(2,1)
60.7
61.3
61.1
(1,2)
62.4
62.2
62.3
(2,4)
62.3
62,6
61.9
(4,2)
60.5
61.6
61.9
(2,2) (Ours SFCNet)
62.9
Table 10: Ablation study on the maximal number of points in spherical frustums on the Se-
manticKITTI validation set.
Maximal Number of Points Spherical Frustum
mloU (%)
2
61.0
4
61.9
Unlimited (Ours SFCNet)
62.9
results further indicate the significance of overcoming quantized information loss in the field of LiDAR point cloud semantic segmentation.
Configuration of the Hash Table. The number of hash functions is the main parameter of the hash table, which means the number of functions used for the hash table retrieval. In the implementation, if the first hash function can successfully retrieve the location of the target point, the other functions will not be used. We change the number hash functions to show the model sensitivity of hash table configurations. As shown in Tab. [1 the performance and inference time of SFCNet have little difference under different numbers hash functions. The results show that in most cases, the first function can successfully retrieve the location, and thus the inference times change slightly in different function numbers. These results indicate that SFCNet is robust to different hash table configurations.
C.4 Comparison of Sampling Methods
We further validate the effectiveness and efficiency of the proposed Frustum Farthest Point Sampling (F2PS) by the qualitative comparison with tride-based 2D sampling and the comparison of time consumption with Farthest Point Sampling (FPS).
Qualitive Comparison. As shown in Fig. Oaa), Stride-Basee 2D Sampling (SBS) only samples the point cloud based on 2D stride. The visualization shows that the stride- sampled point cloud relatively rough. Due to lack of 3D geometric information, SBS fails to sample the 3D point cloud uniformly. Thus, the loss of geometric structure in the sampled point cloud is obvious, such as many broken lines on the ground. Our F2PS takes into account the 3D geometric information based on the FPS in the spherical frustum, which enables F2PS to sample the 3D point cloud uniformly and preserve the significant 3D geometric structure during the sampling.
Time Consumption Comparison. As shown in Fig. b)), with the increment of sampled point number, the cost time of our F2PS increases slowly, while the cost time of FPS increases dramatically, This result shows performing FPS on the frustum point sets is efficient and does not increase the computing burden.
C.5 Comparison between SFCNet and Baseline Model on Small Object Categories
To further show the improvement of small object segmentation, we compare the quantitative results on the small object categories between SFCNet and the baseline model with the KNN-based post- processing [13] on SemT and nuScenes validation sets. As shown in Tabs. [2] and 13 our SFCNet has higher performances on all small object categories compared to the baseline model. The results show overcoming quantized information loss preserves complete geometric information the small objects and thus makes them better recognized and segmented by our SFCNet.
19

---

Table 11: Ablation study on the configuration of the hash table for the spherical frustum structure on the SemanticKrTTI validation set.
Number Hash Functions
Inference Time (ms)
mloU (%)
59.5
62.9
60.1
62.9
59.5
62.9
4 (Ours SFCNet)
59.7
62.9
10
Time
20"
P2PS (Ours)
Original
Strid: Breed glod Point Clood
F2PS Sampled Point Cloud (Ours)
2
Numbee of Points
2
2
2
(a) Qualitative Comparison
(b) Time Consumption Comparison
Figure 6: In this figure, (a) presents the qualitative comparison between stride-based 2D sampling and our Frustum Farthest Point Sampling (F2PS). The red boxes show the zoomed view of the point clouds in the close areas. (b) illustrates the time consumption comparison between Farthest Point Sampling (FPS) and our F2PS.
D More Visualization
To better demonstrate the effectiveness of SFCNet for LiDAR point cloud semantic segmentation, we conduct more visualization on the SemanticKITTI and nuScenes datasets. The results are shown in Figs.
10 and the supplementary video Supplementarylidder mp4.
Qualitative Comparison on NuScenes Validation Set. The results of qualitative comparison between our SFCNet and Range [2]] are shown in Fig. On the nuScenes validation set, SFCNet can also have fewer segmentation emrrrs than Range ViT as the results in the SemanticKITTI dataset. Moreover, the better segmentation accuracy of the 3D small objects, like pedestrians and motorcycles, can also be observed on the nuScenes validation set. The results once more demonstrate semantic segmentation improvement of SFCNet due the overcoming of quantized information loss.
More Qualitative Comparison on SemanticKITTI Test Set. The ground truths on the Se- manticKITTI test set not available. Thus, search for the corresponding RGB image and project the semantic predictions on the image to compare the semantic segmentation accuracy between the state-of-the-arr 2D image-basee based method CENet (20) and SFCNet the SemanticKITTI test set. As shown in Figs. and compared to CENet. SFCNet can more accurately segment the LiDAR point cloud in various challenging scenes on the SemanticKITTI test set.
Specifically, SFCNet recognizes the thin poles distance on the rural road of Fig. 8(a) a) and in the complex intersections of Fig. Ic), while CENet predicts the poles wrong classes. In addition, SFCNet recognizes the thin trunks inside the vegetation on the rural scenes of Fig. ()b) and Fig. b)) while CENet wrongly predicts the trunk the fetch and vegetation respectively. Moreover, SFCNet successfully segments the boxed persons in the complex intersection of Fig. Fig.((c) and in the urban scene of Fig. aa) while CENet gives wrong predictions due to the information loss of the distant persons during 2D projection. These results further validate the better segmentation performance of SFCNet to 3D small objects.
More Qualitative Comparison on NuScenes Validation Set. As the visualization on the Sc- ManticKITTI test set, provide the additional qualitative comparison between our SFCNet and Range ViT on the nuScene validation set with the projected predictions illustrated in Fig. 10 The results further demonstrate the better semantic segmentation of SFCNet for the challenging street scenes on the nuScenes validation set compared to RiTgeViT.
Specifically, in the first scene, the close motorcycle can be correctly segmented by SFCNet, while RangeVil VIT recognizes the motorcycle as a car, which shows that SFCNet can help the autonomous car correctly recognize the type of close obstacles, and enable the car to make appropriate decisions.
20

---

Table 12:
Quantative comparison of semantic segmentation between baseline model and SFCNet for the small object categories on SemanticKlTTI validation sets. Bold results are the best in each column. The performance improvement of each category is highlighted in green.
SemanticKITTI
motorsycle
traliie sign
hicycle
bieyetist
10000000
Approach
Baseline w/ KNN Based Post processing
44.2
46.0
48.8
71.6
73.6
67.4
63.1
45.7
SFCNet (Ours)
44.9(+0.7) 60.6(+1.6) 50.5(+1.7) (3.1(+155 (3.1(+ 68.5 64.6(+1.5) (7.76+1))))788 (7.8(+
Table 13: Quantative comparison of semantic segmentation between baseline model and SFCNet for the small object categories on the nuScenes validation sets. Bold results are the best in each column. The performance improvement of each category is highlighted in green.
nuScenes
motorcycle
podestrion
bicycle
traffic
Approach
Baseline w/ KNN-Based Post-processing
30.6
77.0
73.9
62.8
SFCNet (Ours)
40.4(+9.8) 82.0(+5.0) 78.0(+4.1) 65.8(+3.0)
In the second scene, the distant pedestrians on the other side of the crossing can also be correctly segmented by SFCNet due to the elimination of quantized information loss. In contrast, Range VIT wrongly predicts the pedestrians as traffic cones.
In the third scene, since the boxed pedestrian is close to the manmade, VingeViT confuses it with the manmade and does not segment the pedestrian, while our SFCNet can clearly recognize the boundary and successfully segments the pedestrian.
Sequential Qualitative Comparison SemanticKITTI Validation Set. We demonstrate the qualitative comparison between our SFCNet and the SoTA 2D projection-hasse segmentation meth- ods, CENet and ViT, supplementary video cooplemmntarrVidee .pp4. In this video, the semantic predictions in both the 3D point cloud view and the RGB image view (where the colored point cloud is projected the RGB images) are presented. The results show that our SFCNet can consistently show higher segmentation accuracy on the point cloud of each frame the sequence than 2D projection- methods, which further indicates the stronger semantic segmentation capability of our SFCNet.
continuous sequence the SemanticKITTI validation set in the
21

---

Ground Truth
RangeViT
SFCNet (Ours)
banier
bicycle
bus
car
construtton-
motorcycle
pedestrian
traffic-cone
trailer
truck
driveable-surface
other-flat
sidewalk
terrain
manmade
vegetation
Figure Qualitative Comparison on NuScene Validation Set. In this figure, we conducted the qualitative comparison between Range ViT [2I] and our SFCNet of semantic segmentation on the nuScenes validation show the eтror maps of the predictions of RiT and our SFCNet respectively. In addition, the reference from point color the semantic class the ground truths is shown the bottom. Moreover, the false-see points are marked as red in the error maps. Furthermore, we use circles with the same color to point out the same objects in the ground truth and the two error maps.
The first column presents the ground truths, while the following two columns
22

---

Original RGB Image
Prediction
Projected Prediction
CENet
(a) Urban
SPCNNC (Ours)
CENet
(b) Rural
SECN (Ours)
CENet
(c) Complex Intersections
SPCNN (Ours)
bicycle
molonyyce
truck
vehicle
bicyclist
motonydist
road
parking
sidewalk
otber-ground
buildiag
fence
vegetation
trunk
tamain
pole
traffic sigm
Figure 8: More Qualitative Comparison on Semantic Segmentation on SemanticKITTI Test Set. We
show the qualitative comparison between our SFCNet and the state-of-the-ard 2D Image-asedd method CENet (0) on the SemanticKITTI test set. The visualized challenging autonomous driving scenes include urban, rural, and complex intersection scenes. The predictions projected the corresponding RGB images are also illustrated. In addition, we use the same color boxes to point out the same objects in the point clouds and images for each scene. Meanwhile, we provide the zoomed-in view of some boxed objects for clear visualization. Moreover, the reference from point color to the semantic class in the predictions shown at the bottom of the figure.
23

---

Original RGB Image
Prediction
Projected Prediction
CENe
(a) Urban
SFONE (Our))
CENeE
(b) Rural
SFUNN Oors
CENe
(c) Complex Intersections
SFCNe (Ours)
bicycle
motornycle
truck
other-vehicle
person
hicyclist
motorcccist
road
parking
ghhuunground
building
vegetation
trunk
torain
traffic sign
Figure 9: More Qualitative Results Semantic Segmentation SemanticKITTI Test Set. This figure shows more qualitative results of our SFCNet and the state-of-the-an 2D image-based method
CENet (20] the urban, rural, and complex intersection scenes of the SemanticKITTI test set. As in Fig. g the predictions projected on the corresponding RGB images are also illustrated. In addition, the same color boxes adopted to point out the same objects in the point clouds and images for each scene. Meanwhile, the zoomed-in view of some boxed objects is illustrated for clear visualization. Moreover, the reference from point color to the semantic class in the predictions is shown at the bottom of the figure.
24

---

Ground Truth
RangeViT
SFCNet (Ours)
barrier
bicycle
bus
construction-vehiike
mobonycle
pedestrian
traffic-cone
trailer
truck
driveahle-sune
otber-flat
sidewalk
terrain
manmade
vegetation
Figure 10: More Qualitative Comparison of Semantic Segmentation on NuScenes Validation Set. We show more comparisons between SFCNet and the ofate-of-the-ar 2D image-based method RingeViT [21] on the nuScenes dataset. The predictions projected on the corresponding RGB images also illustrated. In addition, we use the same color boxes to point out the point clouds and images for each scene. Meanwhile, we provide the zoomed-in view of some boxed objects for clear visualization. Moreover, the reference from point color to the semantic class in the predictions and ground truths is shown at the bottom of the figure.
objects in the
25

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly listed the contributions and pointed out the scope of the paper, LiDAR point cloud semantic segmentation, in Sec.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
In the Conclusion (Sec. 5) have discussed the limitations of our work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
26

---

This paper mainly focuses on proposing new data structure for LiDAR point cloud semantic segmentation. No theoretical results are included.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: This paper has included a description of the implementation details for the
experiments in Sec. 4.2 and the additional implementation details in Sec. B of the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
27

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We will release our source code on https//gghhbbbccom///iilaaisStts
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co cc.
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ ctipp.ccc/uublic//gr guwdes/ddeeeee CooeeSumeeeeeeeeeeeeee ssi onyylicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The dataset splits, hyperparameters, and optimizer choice are described in Sec.
and 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification; Some results need to be uploaded to the dataset's official website for evaluation since the ground truths of the test set are not publicly available. Thus, it is not feasible for multiple evaluations to obtain the error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
28

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Please see Sec. 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https://neurips. http:///wwppp.p.ppppptc/eteceeee httpbbiii//hiicccideeeieeeee.e
Answer: [Yes]
Justification: We have made sure that the research in this paper conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The research has no potential negative societal impacts. On the contrary, the work can be applied the autonomous driving field perception and scene understanding.
Guidelines:
The answer NA means that there is no societal impact of the work performed.

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The work is aimed at scene understanding from the LiDAR sensor data. There are no potential risks of misusing.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the papers providing the public datasets used in this paper. The licenses of the datasets will included in the released source code.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
30

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssii paperswi thcode. www.coo//aaaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not include new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
This work is irrelevant to human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This work irrelevant human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
31

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "XbnZOn2VP9", "reviewer": "Reviewer_AnTM", "summary": "This paper introduces a novel spherical frustum structure for LiDAR point cloud semantic segmentation, addressing the quantized information loss in 2D projection-based methods. By preserving all points within frustums and employing a memory-efficient hash-based representation, the Spherical Frustum sparse Convolution Network (SFCNet) achieves superior segmentation accuracy, particularly for small objects. The proposed Frustum Farthest Point Sampling (F2PS) method ensures uniform sampling and computational efficiency. Extensive experiments on the SemanticKITTI and nuScenes datasets demonstrate SFCNet's improved performance over existing methods.", "questions": "I overall like the idea of this paper. However, my concerns are focused on the experimental results. If the authors can address my concerns regarding the experiments, I am inclined to raise my score.", "limitations": "no potential negative social impact", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Innovation in Data Representation**: The spherical frustum structure preserves all points, avoiding quantized information loss and improving segmentation accuracy, especially for small objects.\n- **Memory Efficiency**: The hash-based representation efficiently stores spherical frustums, minimizing memory usage compared to dense grid storage.\n- **Improved Sampling Method**: The Frustum Farthest Point Sampling (F2PS) method ensures uniform point cloud sampling, enhancing the retention of key information while maintaining computational efficiency.", "weaknesses": "- The proposed methods, including the spherical frustum structure and hash-based representation, add layers of complexity that might pose challenges for practical implementation. Interesting to see the analyze on memory usage, parameter size, and inference time.\n- The paper focuses primarily on comparisons with 2D projection-based methods. While it mentions the performance gap with 3D voxel-based methods, a more detailed analysis and direct comparison with state-of-the-art 3D voxel-based approaches would provide a clearer picture.\n- The paper lacks a sensitivity analysis of key parameters, such as the stride size in the F2PS, the number of points in the spherical frustums, and the impact of different hash table configurations, etc. Understanding how sensitive the model performance is to these parameters would be valuable for tuning the model for specific applications.\n- LiDAR data often contains noise and outliers. The robustness of the proposed method to such imperfections is not thoroughly evaluated. Experiments that introduce varying levels of noise and assess the impact on segmentation performance would be beneficial in demonstrating the method’s robustness.\n- The qualitative results, although helpful, are somewhat limited in scope. Providing a more diverse set of visualizations, including different types of scenes (e.g., urban, rural, complex intersections), would offer a clearer picture of the method’s practical performance across various real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable suggestions for our paper. We are encouraged by your confirmation of our data representation, memory-efficient hash-based representation, and efficient point sampling. The following are our responses to your concerns:\n\n---\n### Q1: Analysis of Memory Usage, Parameter Size, and Inference Time.\nPlease refer to the common response to you in Q1 of Author Rebuttal.\n\n---\n### Q2: Detailed Analysis with the 3D Voxel-based Methods.\nPlease refer to the common response to you in Q2 of Author Rebuttal.\n\n---\n### Q3: Sensitivity Analysis of Key Parameters.\nWe have conducted the experiments for different settings of the stride size in the F2PS, the number of points in the spherical frustums, and the configurations of the hash table. \n- **Stride Size in the F2PS:** Due to the computational resource limitation, the ablation studies of four different settings of the stride sizes in the first downsampling layer's F2PS are conducted, including $(1,2),(2,1),(2,4),(4,2)$. Since the point cloud to be downsampled in the first downsampling layer is the densest, the impact of different stride sizes of the F2PS is the most significant. We promise that a thorough analysis will be conducted in the final version of the paper. The results are shown in the table below. The results show for the first F2PS, the $(2,2)$ stride sizes show a better segmentation performance than the other stride size settings. $(2,2)$ stride sizes suitably downsample the point cloud in the vertical and horizon dimensions. Higher or lower downsampling rates result in the oversampling or undersampling of the point cloud respectively. \n\n\t| Stride Sizes $(S_h,S_w)$ | mIoU (%) $\\uparrow$ |\n\t| -| -|\n\t| (2,1) | 60.7|\n\t| (1,2) | 62.4 |\n\t| (2,4) | 62.3  |\n\t| (4,2) | 60.5 |\n\t| (2,2) (Ours SFCNet) | 62.9 |\n- **Number of Points in the Spherical Frustums:**  In the spherical frustum structure, the number of points in the frustum is unlimited and only depends on how many points are projected onto the corresponding 2D location. To analyze the effect of the number of points in the frustum, we set the maximal number of points in each spherical frustum and the points exceeding the maximal point number are dropped. As shown in the table below, preserving more points in the spherical frustum results in better segmentation performance, since more complete geometry information is preserved. These results further indicate the significance of overcoming quantized information loss in the field of LiDAR point cloud semantic segmentation.\n\t\n\t| Maximal Number of Points in Spherical Frustum | mIoU (%) $\\uparrow$ |\n\t| - | -|\n\t| 2 | 61.0|\n\t| 4| 61.9 |\n\t| Unlimited (Ours SFCNet) | 62.9 |\n- **Configuration of the Hash Table:** The number of hash functions is the main parameter of the hash table, which means the number of functions used for the hash table retrieval. In the implementation, if the first hash function can successfully retrieve the location of the target point, the other functions will not be used. We change the number of hash functions to show the model sensitivity of hash table configurations.  As shown in the table below, the performance and inference time of SFCNet have little difference under different numbers of hash functions. The results show that in most cases, the first function can successfully retrieve the location, and thus the inference times change slightly in different function numbers. These results indicate that SFCNet is robust to different hash table configurations.\n\t\n\t| Number of Hash Functions | Inference Time (ms)   $\\downarrow$ | mIoU (%) $\\uparrow$ |\n  | - | - | -|\n\t| 2| 59.5| 62.9 |\n  | 3 | 60.1 | 62.9                |\n\t| 5| 59.5 | 62.9                |\n\t| 4 (Ours SFCNet)  | 59.7| 62.9                |\n\n---\n### Q4: Performance With Different Levels of Noise in LiDAR Data.\nThe gaussian noises with zero mean and different standard deviations, including 0.25m and 0.5m, are added to the original LiDAR point cloud to simulate different levels of noise. As shown in the table below, though the segmentation performance decreases while the level of noise increases, our SFCNet has better performance than the baseline model in all cases. Thus, the proposed SFCNet is more robust to point cloud noises than the baseline model which has quantized information loss.\n\n| Method                       | mIoU (%) $\\uparrow$ |\n|-|-|\n| Baseline w/o noise| 59.7 |\n| SFCNet (Ours) w/o noise| 62.9|\n| Baseline w/ 0.25m noise | 43.8|\n| SFCNet (Ours) w/ 0.25m noise | 46.6|\n| Baseline w/ 0.5m noise| 30.7|\n| SFCNet (Ours) w/ 0.5m noise  | 34.8|\n\n---\n### Q5: Qualitative Results in More Diverse Types of Scenes.\n We show the qualitative results in more diverse types of scenes on the SemanticKITTI test set in Fig. 1 of the attached PDF file, including the urban, rural, and complex intersections.  As shown in Fig. 1, compared to CENet [1], SFCNet can better segment the distant persons in the complex intersections scene of Fig. 1(c-1) and urban scene of Fig.1 (a-2), poles in the urban of Fig. 1(a-1) and complex intersections scene of Fig. 1(c-2), and trunks in the rural scenes of Fig. 1(b-1) and (b-2). These results validate the consistently better segmentation performance of SFCNet in various real-world scenarios, especially for 3D small objects.\n\n---\n### Reference\n[1] H.-X. Cheng _et al._, ‘Cenet: Toward Concise and Efficient Lidar Semantic Segmentation for Autonomous Driving’, ICME, 2022.\n\n---\nWe hope our response can address your concerns about the experimental results. If you have further problems, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gxO1mLCIbu", "reviewer": "Reviewer_bPaa", "summary": "The paper introduces a Spherical Frustum Sparse Convolution Network (SFCNet), a novel approach for LiDAR point cloud semantic segmentation that addresses the quantized information loss existing in 2D projection-based methods. By utilizing a spherical frustum structure, the SFCNet preserves all points within a 2D projection area, thus maintaining the complete geometric structure of the data. This model improves the segmentation accuracy, particularly for small objects, through the innovative use of hash-based spherical frustum representation and a custom sampling method called Frustum Farthest Point Sampling (F2PS). Extensive evaluations on the SemanticKITTI and nuScenes datasets demonstrate superior performance over existing methods.", "questions": "Q1a. Performance Change on Large Classes. Could the authors provide insights into why SFCNet shows slightly weaker performance on wide-range classes, such as roads and parking areas (see also W1)?\n\nQ1b. Maintaining performance on large Classes. Are there any thoughts how the model could improve the small classes while maintaining performance on larger classes?\n\nQ2. Performance Comparison toward tail-end of Distribution. Given a different performance w.r.t. class sizes, additional tests on accuracy w.r.t. data changes might be helpful. How does SFCNet adapt its segmentation accuracy in non-standard environments (like in night settings, heavy rain, etc.)? Insights here could greatly influence its applicability in real-world settings.", "limitations": "Investigating how the system performs in unpredictable, non-standad settings would offer a clearer picture of its real-world readiness. The paper can also benefit from a deeper exploration of SFCNet's slightly weaker performance on wide-range classes such as roads and parking areas. Understanding the factors that contribute to these discrepancies could lead to improvements in the model's overall accuracy and applicability in real-world scenarios.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. Originality of Spherical Frustum Structure. The spherical frustum structure is a novel approach that effectively preserves all the points within a given 2D projection area, mitigating the common issue of quantized information loss.\n\nS2. Efficiency of Representation. The proposed hash-based spherical frustum representation ensures memory efficiency and computational feasibility.\n\nS3. Performance Boost. The enhanced accuracy in segmenting small objects and maintaining geometric integrity significantly contributes to the field, especially for applications in autonomous driving and robotic navigation.", "weaknesses": "W1. Performance Decrease on Large Classes. While SFCNet introduces significant advancements in semantic segmentation of LiDAR data, the model demonstrates slightly weaker performance on wide-range classes, such as roads and parking areas. This suggests that while the model excels in handling small object segmentation, its techniques might not be as effective for broader landscapes or larger objects.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable suggestions. We are encouraged by your confirmation of our spherical frustum structure, efficiency, and performance improvement on small objects. The following are our responses to your concerns:\n\n---\n### Q1: Insights and Improvement Direction on Performance of Wide-Range Classes.\n- **Discussion on Performance:** To implement the 2D convolution on the spherical frustum, only the nearest points in the neighbor spherical frustums are adopted in the spherical frustum sparse convolution. This design may limit the receptive field of the network and thus result in a slightly weaker performance of the wide-range classes. \n- **Improvement Direction:**  The performance improvement on small classes mainly results from the proposed spherical frustum structure, which overcomes quantized information loss and preserves the complete geometry structure. Thus, to maintain the performance on both the wide-range classes and small classes, the direction is to expand the receptive field based on our spherical frustum structure. To realize this, future work can lie in combining the vision network architecture with a larger receptive field, like the vision transformer [1] or vision mamba [2], with our spherical frustum structure.\n\n---\n### Q2: Additional Tests on Accuracy w.r.t Data Changes.\nWe adopt two methods to change the data size, including increasing instance class sizes by instance augmentation [3] and decreasing labeled class sizes using the ScribbleKITTI [4] dataset. The number of instances before and after increasing instance class sizes are shown in the first table below. In addition,  the second table below shows how many labeled million points belong to each class in the original and scribble datasets.\n|| car    | bicycle | motorcycle | truck | other-vehicle | person | bicyclist | motorcyclist |\n| -| ------ | ------- | ---------- | ----- | ------------- | ------ | --------- | ------------ |\n| Before Increasing | 149531 | 4231    | 2835       | 2543  | 7072          | 7287   | 1390      | 532|\n| After Increasing  | 149575 | 5809    | 4387       | 5846  | 8433          | 8208   | 5437      | 4535         |\n\n\n|| car  | bicycle | motorcycle | truck | other-vehicle | person | bicyclist | motorcyclist | road  | parking | sidewalk | other-ground | building | fence | vegetation | trunk | terrain | pole | traffic-sign |\n| -------- | ---- | ------- | ---------- | ----- | ------------- | ------ | --------- | --| ----- | ------- | -------- | ------------ | -------- | ----- | ---------- | ----- | ------- | ---- | ------------ |\n| Original | 99.4 | 0.392   | 0.94       | 4.59  | 5.46          | 0.817  | 0.299     | 0.088        | 467.1 | 34.6    | 338.2    | 9.17         | 311.8    | 170.0 | 627.2      | 14.2  | 183.6   | 6.71 | 1.44         |\n| Scribble | 6.57 | 0.07    | 0.18       | 0.24  | 0.43          | 0.16   | 0.05      | 0.02         | 16.1  | 1.93    | 24.7     | 0.895        | 25.5     | 38.5  | 60.5       | 2.76  | 9.15    | 1.35 | 0.27         |\n\nThe experiments are conducted on the two cases. As shown in the table below, increasing the instance class sizes can improve the segmentation performance, since more instances of the rare classes are added to ease the long-tailed problem. As for decreasing the labeled class sizes, the performance also decreases since the supervision is weaker with fewer labels.\n\n| Method| mIoU (%) $\\uparrow$ |\n| -| -|\n| SFCNet (Ours)| 62.9                |\n| SFCNet (Ours) w/ increment of instance class sizes | 63.2                |\n| SFCNet (Ours) w/ decrease of whole class sizes     | 56.0                |\n\n---\n### Q3: How SFCNet Adapts Accuracy in Non-Standard Environments.\n\nRobo3D [5] dataset provides the LiDAR point cloud adding simulated noises in non-standard environments and corresponding semantic labels on the SemanticKITTI validation set. However, the night and heavy rain environments are not provided by Robo3D. We do not find any other datasets that provide the data in the night and heavy rain environments modified from the SemanticKITTI dataset. Thus, to fairly test how our SFCNet and the baseline model adapt accuracy in the non-standard environments with the same point cloud data, the snow and wet ground environments on the Robo3D dataset are adopted. Actually, the snow and wet ground are similar to the dropping rain and wet ground in the heavy rain environment. Meanwhile, the night environment does not cause noise in the LiDAR point cloud since the LiDAR sensor captures the range information by actively sending the LiDAR ray. As shown in the table below, when tested on noisy non-standard environments, the performance of our SFCNet decreases. However, it still has a better performance than the baseline model due to overcoming the quantized information loss. These results indicate the robustness of our method to overcome quantized information loss in various real-world settings.\n\n| Method        | mIoU (%) $\\uparrow$ (Snow) | mIoU (%) $\\uparrow$ (Wet Ground) |\n| ------------- | -------------------------- | -------------------------------- |\n| Baseline      | 44.2| 53.9|\n| SFCNet (Ours) | 47.5| 55.8|\n\n---\n### Reference\n[1] Z. Liu _et al._, ‘Swin transformer: Hierarchical vision transformer using shifted windows’, ICCV, 2021.\n\n[2] L. Zhu _et al._, ‘Vision mamba: Efficient visual representation learning with bidirectional state space model’, ICML, 2024.\n\n[3] Z. Zhou _et al._, \"Panoptic-polarnet: Proposal-free lidar point cloud panoptic segmentation.\", CVPR, 2021.\n\n[4] O. Unal _et al._, ‘Scribble-supervised lidar semantic segmentation’, CVPR, 2022.\n\n[5] L. Kong _et al._, ‘Robo3d: Towards robust and reliable 3d perception against corruptions’, ICCV, 2023.\n\n---\nWe hope our response can address your concerns about the performance on non-standard environments, accuracy testing with data changes, and the reasons and improvement direction on the performance of wide-range classes. If you have further problems, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ylhi5XYOoP", "reviewer": "Reviewer_MsCy", "summary": "This paper introduces SFCNet, a spherical frustum sparse convolution network designed for semantic segmentation of LiDAR point clouds. Traditional 2D projection methods suffer from quantized information loss when multiple points project onto the same pixel, leading to sub-optimal segmentation. To address this, the authors propose a spherical frustum structure that retains all points within the same pixel by using a hash-based representation. This structure is then utilized in spherical frustum sparse convolution, which considers both neighboring frustums and the nearest point within each frustum for convolution. Additionally, Frustum Farthest Point Sampling (F2PS) is used to sample points stored in spherical frustums. Experimental results on the SemanticKITTI and nuScenes datasets demonstrate that SFCNet outperforms conventional 2D projection-based methods, particularly in the segmentation of small objects, while maintaining complete point cloud information.", "questions": "In Table 1, it is clear  that 3D voxel-based methods achieve significantly better results compared to point-based and 2D projection-based methods. Does this suggest that 3D voxel-based methods are inherently superior?\n\nCould you provide a comparison of the computational efficiency of SFCNet against other baseline methods? This information is crucial to evaluate the practical applicability and performance trade-offs of SFCNet.", "limitations": "No limitations are discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow. The introduction to the Spherical Frustum Sparse Convolution and Frustum Farthest Point Sampling is clear and comprehensible. It provides a clear narrative structure supported by comprehensive figures and tables. The experimental settings are reasonable and well-documented.\n2. The motivation to address quantized information loss during projection is well-founded.", "weaknesses": "1. The experimental results do not demonstrate a clear superiority of the proposed method. While the SFCNet shows some mIoU improvements over prior projection methods on the SemanticKITTI and nuScenes datasets, these gains are marginal. Furthermore, the performance of SFCNet is still significantly below that of SOTA 3D voxel-based methods.\n2. The paper does not provide an analysis of the computational efficiency of the proposed method. Given that SFCNet involves specific point sampling for convolution, assessing the computational efficiency is crucial to understand its practical significance and potential trade-offs in real-world applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable suggestions. We are encouraged by your confirmation of the presentation of our paper and our motivation for overcoming quantized information loss. The following are our responses to your concerns:\n\n---\n\n### Q1: Performance Gain Compared to Prior 2D Projection-based Methods.\n\nThough the performance gains of the mIoU metric are marginal, SFCNet achieves significant performance improvement on small objects compared to the previous 2D projection-based methods as shown in Tables 1 and 2 in the main manuscript, especially on the human, motorcycle, etc. The improvement mainly results from overcoming quantized information loss by the proposed spherical frustum structure. The better performance of small objects shows the significant practical value of SFCNet for safe autonomous driving.\n\n---\n\n### Q2: Performance Gap between SFCNet and 3D Voxel-based Methods.\nAs shown in the common response to you in Q2 of Author Rebuttal, SFCNet can realize the trade-off between efficiency and performance and thus has a realistic value in specific scenarios. In addition to the realistic contribution, SFCNet also builds a foundation for future works on the 2D projection-based methods since our work provides the spherical frustum structure to overcome the quantized information loss during 2D projection. At the beginning of deep learning-based point cloud semantic segmentation, the 2D projection-based method is the pioneer that shows reasonable performance for semantic segmentation of outdoor LiDAR point cloud since it can transfer the success of image semantic segmentation to the field of point cloud semantic segmentation. Therefore, though 3D voxel-based methods have better performance than 2D projection-based methods recently, it does not mean the 2D projection-based method is inherently weaker than 3D voxel-based network. If a strong image semantic segmentation backbone is proposed, the proposed spherical frustum structure can be adopted to apply the strong backbone to the field of LiDAR point cloud semantic segmentation without quantized information loss. \n\n---\n\n### Q3: Analysis of Computational Efficiency.\nAs shown in the common response to you in Q1 of Author Rebuttal, SFCNet shows better computational efficiency against the other baseline methods. Though we have a specific point sampling method, the computational complexity of our point sampling method Frustum Farthest Point Sampling (F2PS) is linear as shown in the L203 in the main manuscript. Thus, both the theoretical and experimental analysis show our method is computationally efficient and shows significant potential for practice application.\n\n---\nWe hope our response can address your concerns about the performance and computational efficiency between our SFCNet and the other baseline methods. If you have further problems, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "we0T7CzJfw", "reviewer": "Reviewer_BSNT", "summary": "This paper proposes a Spherical Frustum Sparse Convolution Network to address the challenge of LiDAR point cloud semantic segmentation. Traditional approaches often project point clouds into 2D images and apply 2D convolutional neural networks (CNNs) or vision transformers, leading to quantization information loss due to the overlap of multiple points projected onto the same 2D location. To overcome this limitation, the paper introduces a novel spherical frustum structure that allows for direct processing of the 3D point cloud data, preserving spatial and geometric information. The proposed Spherical Frustum Sparse Convolution Network achieves more accurate semantic segmentation of LiDAR point clouds, offering a promising approach for robot perception and scene understanding.", "questions": "Despite the fact that the paper is full of content, it always gives a discomfort in the presentation. Obviously, the authors are requested to resize Figure 1, remove the captions in the figures of Figures 2 and 3 and optimize their presentation. It is recommended that the internal spacing of Table 1 be increased appropriately. The authors are invited to review more carefully.\n\nIn addition, similar to the work on semantic segmentation of large scenes, the authors should compare it with state-of-the-art methods such as DRQNet (ACMMM'2023) and JoSS (INFFUS'2024).", "limitations": "This paper proposes a method SFCNet for point cloud semantic segmentation by directly projecting 3D point clouds into 2D spherical frustums, the main limitation of which lies in the large performance gap. Of course, the proposed method still has a large potential in specific scenarios.", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **Direct 3D Processing**: The Spherical Frustum Sparse Convolution Network processes the LiDAR point cloud data directly in 3D space, preserving the spatial and geometric information of the points. This avoids the information loss that occurs in traditional 2D projection-based methods.\n2. **Reduced Quantization Loss**: By operating directly on the 3D point cloud, the proposed network significantly reduces quantization loss compared to projection-based approaches. This leads to more accurate semantic segmentation results.\n3. **Efficient Sparse Convolution**: The network employs sparse convolution, which is tailored for sparse data such as point clouds. This allows for efficient computation and reduced memory usage, enabling the network to handle large-scale point cloud data.\n4. **Spherical Frustum Structure**: The novel spherical frustum structure enables the network to capture contextual information from different orientations and distances, enhancing its ability to identify objects and regions in the point cloud.\n5. **Improved Performance**: Experimental results show that the proposed Spherical Frustum Sparse Convolution Network achieves superior performance compared to existing methods on various LiDAR point cloud datasets, demonstrating its effectiveness for semantic segmentation tasks.\n6. **Flexible and Extensible Framework**: The network framework is flexible and can be easily extended to incorporate additional components or techniques, providing opportunities for further improvements and adaptability to different applications.", "weaknesses": "1. **Lack of Conceptual Novelty**: The use of spherical structures, while effective, appears to have been partially explored in SphereFormer. This limits the novelty of the proposed SFCNet in terms of the core concept.\n\n2. **Performance Gap**: Based on the information provided, the performance of SFCNet for semantic segmentation of LiDAR point clouds lags far behind that of SphereFormer, whose radial window transformer structure may achieve higher recognition accuracy, especially for distant objects.\n\n3. **Potential for Further Optimization**: Since SFCNet based on 2D projections lags behind SphereFormer based on 3D voxels, there may be room for optimization in terms of data input or network architecture. This paper does not discuss potential future avenues to address this performance gap or to work on different application scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your valuable suggestions for our paper. We are encouraged by your confirmation of the information preservation, efficiency, contextual information capture ability, performance improvement, network flexibility, and the **\"large potential in specific scenarios\"** of our SFCNet. The following are our responses to your concerns:\n\n---\n\n### Q1: Conceptual Novelty.\n\nThe conceptual novelty of our proposed method is solving the quantized information loss during the 2D projection. To address the challenge, we propose the spherical frustum structure to preserve all the points projected onto the same 2D location and the Spherical Frustum sparse Convolution (SFC) and Frustum Farthest Point Sampling (F2PS) to ensure efficiency under the representation of spherical frustum structure. In contrast, the conceptual novelty of SphereFormer [1] is expanding the receptive field of long-range objects by proposing the radius window transformer. Thus, our conceptual novelty is different from SphereFormer.\n\n---\n\n### Q2: Performance Gap with SphereFormer.\n\nThe performance gap between our SFCNet and SphereFormer mainly results from the different representations of the point cloud. SphereFormer adopts the 3D voxel representation and utilizes the 3D convolutional backbone to obtain effective 3D features of the point cloud. Based on the 3D features, it proposes the radius window transformer to further expand the receptive field and enhance the segmentation performance. In this paper, we mainly focus on solving the quantized information loss of the 2D projection-based methods. Thus, we conduct the experiments based on the 2D convolutional backbone, which results in the performance gap with SphereFormer based on the 3D convolutional backbone. However, we realize a smaller performance gap with the 3D methods compared to the other 2D projection-based methods.\n\n----------\n\n### Q3: Future Avenues for Addressing Performance Gap.\n\nIn this paper, we address the quantized information loss for the 2D projection-based methods. Thus, our method provides the information-lossless mechanism for future work to apply the latest research results of image feature learning to the field of LiDAR point cloud semantic segmentation. The future avenues to narrow the performance gap with 3D voxel-based methods are combining the latest image feature learning network architecture, like vision mamba [2], with our information-lossless spherical frustum structure.\n\n----------\n\n### Q4: Addressing the Discomfort in the Presentation.\n\nWe apologize for causing your discomfort due to our representation. We will polish our presentation according to your valuable suggestions. Specifically, we will resize Figure 1 to a suitable size, remove the captions inside Figures 2 and 3, adjust the contents of Figures 2 and 3, and increase the internal spacing of Table 1 in the final version of our paper.\n\n----------\n\n### Q5: Comparison with the State-of-The-Art Methods DRQNet and JoSS.\n\nWe compare our SFCNet with the state-of-the-art methods DRQNet [3] and JoSS [4] as follows. We will add these comparisons and discussions in the final version of our paper.\n\n-   **DRQNet:** DRQNet proposes the dual representation query strategy and representation selective dependency module for weakly-supervised LiDAR point cloud semantic segmentation. The proposed modules of DRQNet improve the feature aggregation and fusion in the dual feature space of the point cloud and construct a stronger self-supervised signal for weakly-supervised learning. Compared to DRQNet, our method mainly focuses on proposing an efficient information-lossless data representation and supervised learning method of geometric information-only LiDAR point cloud. The weakly-supervised learning architecture based on the spherical frustum structure is one valuable direction for our future work.\n    \n-   **JoSS:** JoSS proposes the cross-modal transformer-based feature fusion method to adopt the cross-attention mechanism for better information fusion. In addition, unimodal data augmentation is proposed in JoSS for point-level contrastive learning. Compared to JoSS, SFCNet does not adopt the RGB image for multi-modal feature fusion and 3D segmentation backbone for point cloud feature extraction. Thus, SFCNet shows a weaker segmentation performance compared to JoSS on the SemanticKITTI test set. However, SFCNet overcomes the quantized information loss of the 2D projection-based method and shows potential to be adopted in applying the future stronger 2D backbone to the field of LiDAR point cloud semantic segmentation.\n    \n\n----------\n\n### Reference\n\n[1] X. Lai _et al._, ‘Spherical Transformer for LiDAR-based 3D Recognition’, CVPR, 2023.\n\n[2] L. Zhu _et al._, ‘Vision mamba: Efficient visual representation learning with bidirectional state space model’, ICML, 2024.\n\n[3] J. Liu _et al._, ‘Exploring Dual Representations in Large-Scale Point Clouds: A Simple Weakly Supervised Semantic Segmentation Framework’, ACMMM, 2023.\n\n[4] Y. Wu _et al._, ‘Joint Semantic Segmentation using representations of LiDAR point clouds and camera images’, Information Fusion, 2024.\n\n----------\n\nWe hope our response can address your concerns about the conceptual novelty, performance gap with SphereFormer, and presentation discomfort. If you have further problems, please let us know.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We appreciate all reviewers' efforts and valuable suggestions. We are grateful to receive the positive comments from the reviewers, including memory efficiency (Reviewer AnTM, bPaa, BSNT), innovation in spherical frustum structure (Reviewer AnTM, bPaa), and performance "boost" (Reviewer bPaa, BSNT). 

We have responded to the concerns of each reviewer individually. In addition, we also notice the common concerns, including memory and computational efficiency (Reviewer AnTM, MsCy), and performance gap with 3D voxel-based methods (Reviewer AnTM, MsCy), raised by the reviewers. Thus, we provide the common response to these concerns as follows:

---

### Q1:  Analysis of Memory Usage, Parameter Size, and Inference Time (Reviewer AnTM, MsCy).
As shown in the table below, the analysis of the memory usage, parameter size, and inference time are conducted through the comparison between our SFCNet and the other baselines. The normalized memory usage and inference time, computed by dividing the whole memory usage and inference time by the number of thousand points, are also shown. 
The results show:

- **Memory Usage:** From the metric of normalized memory usage, the memory efficiency of our SFCNet is better than the point-based methods [1-2] and 3D voxel-based methods [3-4]. As for the 2D projection-based methods, SFCNet has a better memory efficiency than the baseline model and a comparable memory efficiency with RangeViT [5]. The results show that based on memory-efficient hash-based representation, the spherical frustum structure introduces little extra memory consumption and is suitable for practical application.
- **Parameter Size:** Since SFCNet has pure 2D convolutional layers, SFCNet has a much smaller parameter size than the transformer-based methods [4-5] and 3D convolution-based methods [3]. In addition, the comparison between SFCNet and the baseline shows the spherical frustum structure does not introduce extra parameters. As for the comparison with the MLP-based methods [1-2], though the parameter size of SFCNet is larger than the MLP-based methods, SFCNet has better segmentation performance.
- **Inference Time:** Our SFCNet has the smallest normalized inference time among the compared methods. The results show the spherical frustum structure and hash-based representation are also computationally efficient. 

| Method           | Points | Memory Usage (M) $\downarrow$ | Normalized Memory Usage (M/K) $\downarrow$ | Parameter Size（M）$\downarrow$ | Inference Time (ms) $\downarrow$ | Normalized Time (ms/K) $\downarrow$ |
| -| -| -| -| -| -| -|
| PointNet++ [1]   | ~45K   | 841                           | 18.69                                      | 3.8  | 131.0 | 2.91                                |
| RandLA [2]       | ~120K  | 1395                          | 11.63                                      | 4.9                             | 212.2                            | 1.77                                |
| Cylinder3D [3]   | ~40K   | 1912                          | 47.80                                      | 214                             | 67.5                             | 1.69                                |
| SphereFormer [4] | ~90K   | 2678                          | 29.76                                      | 124                             | 108.2                            | 1.20                                |
| RangeViT [5]     | ~120K  | 1344                          | 11.20                                      | 91                              | 104.8                            | 0.87                                |
| Baseline         | ~90K   | 1090                          | 12.11                                      | 44                              | 46.4                             | 0.52                                |
| SFCNet (Ours)    | ~120K  | 1386                          | 11.55                                      | 44                              | 59.7                             | 0.49                                |

---

### Q2: Detailed Analysis of Performance Gap with 3D Voxel-Based Methods. (Reviewer AnTM, MsCy).

The 3D voxel-based methods adopt the 3D convolutional kernel to learn the semantic mode in the 3D space. In contrast, SFCNet learns the semantic mode through the 2D convolutional kernel, which has a weaker ability to learn the 3D semantic mode and results in a weaker performance of SFCNet than the 3D voxel-based methods. However, based on the efficiency analysis in Q1, compared to the 3D voxel-based methods, our SFCNet has better computational efficiency, lower memory consumption, and fewer parameters. Thus, though there exists a performance gap between our SFCNet and the 3D voxel-based methods, our SFCNet can realize a trade-off between efficiency and performance and has the potential to be applied in specific scenarios.

---

In addition, we provide an attached PDF, where we show the qualitative results in more diverse scenes in Figure 1 for Reviewer AnTM.

---

### Reference
[1] C. R. Qi _et al._, ‘Pointnet++: Deep hierarchical feature learning on point sets in a metric space’, NeurIPS, 2017.

[2] Q. Hu _et al._, ‘Learning semantic segmentation of large-scale point clouds with random sampling’, TPAMI, 2021.

[3] X. Zhu _et al._, ‘Cylindrical and asymmetrical 3d convolution networks for lidar segmentation’, CVPR, 2021.

[4] X. Lai _et al._, ‘Spherical Transformer for LiDAR-based 3D Recognition’, CVPR, 2023.

[5] A. Ando _et al._, ‘RangeViT: Towards Vision Transformers for 3D Semantic Segmentation in Autonomous Driving’, CVPR, 2023.

Meta-review: After rebuttal and discussion, all four reviewers are positive, but to a varying degree: all scores between borderline accept and strong accept were given. The main strengths are novelty of the representation, the efficiency, and the performance, where the least positive reviewer criticizes the lack of experimental evidence on the latter.
Decision: Accept (poster)