GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation
Authors: Haibo Qiu
Baosheng Yu
Dacheng Tao
Venue: Accepted by TMLR ()

Abstract: Point cloud semantic segmentation from projected views, such as range-view (RV) and bird's-eye-view (BEV), has been intensively investigated. Different views capture different information of point clouds and thus are complementary to each other. However, recent projection-based methods for point cloud semantic segmentation usually utilize a vanilla late fusion strategy for the predictions of different views, failing to explore the complementary information from a geometric perspective during the representation learning. In this paper, we introduce a geometric flow network (GFNet) to explore the geometric correspondence between different views in an align-before-fuse manner. Specifically, we devise a novel geometric flow module (GFM) to bidirectionally align and propagate the complementary information across different views according to geometric relationships under the end-to-end learning scheme. We perform extensive experiments on two widely used benchmark datasets, SemanticKITTI and nuScenes, to demonstrate the effectiveness of our GFNet for project-based point cloud semantic segmentation. Concretely, GFNet not only significantly boosts the performance of each individual view but also achieves state-of-the-art results over all existing projection-based models. Code is available at \url{https://github.com/haibo-qiu/GFNet}.

Published in Transactions on Machine Learning Research (09/2022)
GFNet: Geometric Flow Network for 3D Point Cloud Semantic Segmentation
Haibo Qiu
School of Computer Science, The University of Sydney, Australia
Baosheng Yu
wttpheengg.u@@@yydeyy...o edw..
School of Computer Science, The University of Sydney, Australia
Dacheng Tao
dacheng. tao Bsydney. edu.
School of Computer Science, The University of Sydney, Australia
Reviewed on OpenReview: https: // hptp://eepeeveee.eeem net/ harmm? LSALSAY5s
Abstract
Point cloud semantic segmentation from projected views, such as range-view (RV) and bird's- eye-view (BEV), has been intensively investigated. Different views capture different informa- tion of point clouds and thus are complementary to each other. However, recent projection- based methods for point cloud semantic segmentation usually utilize a vanilla late fusion strategy for the predictions of different views, failing explore the complementary informa- tion from geometric perspective during the representation learning. In this paper, we in- troduce geometric flow network (GFNet) to explore the geometric correspondence between different views in an align- align-before-fur before-fuse manner. Specifically, devise novel geometric flow module (GFM) to bidircctionally align and propagate the complementary information across different views according to geometric relationships under the end-to-end learning scheme. We perform extensive experiments on two widely used benchmark datasets, SemanticKITTI and nuScenes, to demonstrate the effectiveness of our GFNet for project-based point cloud semantic segmentation. Coneretely, GFNet not only significantly boosts the performance of each individual view but also achieves state-of-the-arl results over all existing projection- based models. Code is available at https:///gicubb.cc/naaloooo/////
1
Introduction
3D point cloud analysis has drawn increasing attention from both academic and industrial communities, since the wide deployments of lidar sensors have made possible to obtain abundant 3D point cloud data (Behley) et al. 2019 Caesar et al. 2020 Compared to 2D images (e.g., RGB images), a point cloud can capture precise structures of objects, thus providing a geometry-accurate perspective representation, intrinsically in line with the 3D real world. Point cloud semantic segmentation, aiming to assign semantic label to each point, is fundamental to scene understanding, which enables intelligent agents to precisely perceive not only the objects but also the dynamically changing environment. Therefore, point cloud semantic segmentation plays a crucial role, especially safety-critical applications such as autonomous driving LLi et al. 2020: Aksoy et al. 2020) and robotics (Li et al.) 2019 Yang et al. 2020
Unlike structural pixels in image, point cloud is set of points represented by (z, 2, coordinates with- out specific order, and extremely sparse for in-the-wild scenes. Hence, it is non-trivial to utilize off-tho-shelf deep learning Lechnologies on images for point cloud analysis. Recent point cloud segmentation mel hods usually address the mentionel sparse distributed issue from the perspectives of either voxelization, single/multi-view projections, or novel point-bascd operations. However, voxcl-based methods mainly suffer from heavy computations while point- operations struggle to efficiently capture the neighbour infor-
mation, especially when dealing with large-scale outdoor scenes (Behley et al.] 2019 Caesar et. al. 2020

---

Published in Transactions on Machine Learning Research (09/2022)
With the great success of fully convolutional networks for Image-based semantic segmentation (Chnn et al, 2017a Long et al. 2015 Zhao et. al. 2017). projection-based methods have recently received increasing attention. Figure illustrates two widely used projected views, i.e., range-view (RV) (Milioto et al. 2019 and siid (BEV) (Zhang ct al. 2020b). Single view based methods can only learn vicw-specific
represenlations (Alonso et al. 2020 Cortinhal et al. 2020 Xu et al. 2020 failing to handle those occluded
points during projection. For example, the RV in Figure 2 shows occluded tail phenomenon (i.e., the
distant occluded points are assigned with the labels of near displayed points) in the red rectangle areas. To address this problem, recent methods resort to multi-view models to incorporate complementary information over different views, which usually deal with RV /BEV in sequence (Chen et al. 2020 Gerdzhev et. al. 2021
or perform vanilla late fusion (Alnaggar et al. 2021 Liong et al., 2020). However, existing methods fail to probe the intrinsically geometric connections of RV /BEV during the represenlation learning.
As we can see from Figure [ to find the geometrie correspondence between two views (the dash line), we can utilize the original point cloud as bridge, e.g., the transformation RV to BEV can be ob- tained from two transformations (the solid line): 1) from RV to point cloud; and 2) from point cloud to BEV. Inspired by this, we introduce a novel geomet- ric flow network (GFNet) to simultaneously learn view-specific representations and explore the geo- metric correspondences between RV and BEV in end-to-end learnable manner. Specifically, propose to adopt two branches to process RV and BEV imputs, where each branch follows an encoder- decoder architecture using a ResNet et al. 2016) as the backbone. We then devise a geometric flow module (GFM), which then applied at multiple levels of feature representations, to bidirectionally align and propagate geometric information across two poojection views, aiming to learn more discrim- inative representations. Figure illlssrates an ex- ample of propagating the information from BEV to RV which benefits handling those occluded points by RV projection. In addition, inspired by Kochanov et al. (2020), we also use KPConv (Thomas et al. 2019 at the top of GFNet to replace a KNN post-processing, thus making it easy to train the overall multi-view point cloud semantic segmentation pipeline in an end-to-end paradigm. The main contributions of this paper are summarized as follows:
Spherical Proj
RV
Back Proj
RranB
Boo- Trans
first
BEV
Back
Top-Down Proj
Figure 1: Geometric bidirectional transformation di- agram between range-view (RV) and bird's s-eee-view (BEV).
We introduce a novel GFNet to simultaneously learn and fuse multi-view representations, where the proposed geometric flow module (GFM) enables the geometric correspondence information to flow across different views.
We devise two branches for RV and BEV with KNN post-processing replaced by KPConv, making the proposed GFNet end-to-end trainable.
Extensive experiments are performed on two popular large-scale point cloud semantic segmentation benchmarks, i.e., SemanticKITTI and nuScenes, demonstrate the effectiveness of GFNet, which achieves state-of-the-art performance over all existing projection-based models.
2
Related Work
In this section, we review recent point cloud semantic segmentation literature from the perspectives point- basod, voxol-basod, and projection-based methods. In addition, we discuss more recently hybrid methods which simultancously use multiple crmmats/modllttice Among all projection- based methods, we mainly focus the multi-view projection-based methods.
2

---

Published in Transactions on Machine Learning Research (09/2022)
car
blcydist
road
parking
sidewalk
RV
BEV
other-ground
vegetation
trunk
terrain
pole
traffic-sign
GT
GFNet
The distant occluded points caused by RV projection are misclassified as the labels of near displayed points in the red rectangle arcas, while they are totally captured by BEV. By propagating the information between BEV and RV, this issue can be well addressed by our GFNet.
Figure 2:
Point-based Mcthods. Recent methods mainly devise novel point operations/architoctuuss to directly
learn representations from raw points (Hu et al. 2020 Li et al. 2018
Qi et al. 2017a b) Thomas et al. 2019
Wang et al. 2019 Tang et al. 2022), including mlp- based (Hu et al. 2020 Qi et al. 2017a b). enn-based (Li
ct al.
2018): and graph-based methods (Thomas et al. 2019 Wang et al. 2019). Specifically, PointNet (Qi
ct al. 2017a a pioncer that directly processes point cloud with multi-layer perceptron (MLP), which 2017b) using hierarchical neural network to learn local features. PointCNN (Li al. 2018 learns X-transformation from the input points for alignment, followed by typical convolution layers. DGCNN (Wang et al.
is improved by Not+ (Qi et al.
2019) proposes a new graph convolution module called EdgeConv to capture local geometric features. LAndLA-Net (Hu el al. 2020) employs random point sampling with an effective local feature aggregation module to persevere the local information. KPConv (Thomas et al. 2019) introduces a new point convolution operator named Kernel Point Convolution to directly take neighbouring points as input and processes with spatially located weights. Nevertheless, the irregular and disordered characteristics of point clouds make it inefficient to capture the neighbour information.
Voxel-based Methods. They (Cheng et al. 2021 Tang et al. 2020 Yan et al. 2020a Zhang et al. 2020a; Zhu et al. 2021) | first voxelize point clouds to regular grids and process with 3D convolutions. Cylin-
der3D ZZu et al.) 2021) introduces the cylindrical partition and asymmetrical 3D convolution networks to tackle the issues of sparsity and varying density of point clouds. SPVNAS (Tang et al. 2020) proposes Sparse Point-Voxel Convolution (SPVConv), which is a lightweight 3D module consisting the vanilla Sparse Con- volution and the high-resolution point- base- branch. Furthermore, 3D Neural Architecture Search (3D-NAS) is presented to obtain the efficient and effective architecture for semantic segmentation. AF2S3Net (Cheng) et al. 2021 designs AF2M to capture the global context and local details and an AFSM to learn inter- relationships between channels across multi-scale feature maps from AF2M. However, the distributions of large-scale outdoor scenes (e.g., SemanticKITTI (Behley et al. putations grow cubically when increasing the voxel resolution.
2019 are extremely sparse, and the com-
Yan al. 2022 usually focus on simultaneously using multiple formats/ moddaiiies e.g., voxel, points or natural images) to learn discrimina- tive representations. DRINet++ Ye et al. 20221) proposes Sparse Feature Encoder to extract local context information from voxelized grids, and Sparse Geometry Feature Enhancement to enhance the geometric char- acteristics of sparse points using multi-scalc sparse projection and attentive multi-scale fusion. RPVNet Xu eet al. (2021) explores multiple and mutual information interactions among three views (i.e., projection, voxel and point), following by a gated fusion module to adaptively merge the three features based on concurrent inputs. 2DPASS Yan et al. (2022) assists raw points with 2D natural images. It distills richer semantic and structural information from 2D images without strict paired data constraints to the pure 3D point network, by leveraging an auxiliary modal fusion and multi-scale fusion-to-single knowledge distillation (MSFSKD).
Hybrid Methods. Recent methods (uu et al., 2021 Ye al. 2021
3

---

Published in Transactions on Machine Learning Research (09/2022)
Geometric Flow Module
K.PComv
GS
Grid Sample
Loss Computing
tRV
KIX
M,
Fr
M
BEV
b
M: M, or M
GS
Fr or Fb
6S
KPC
M
F
HAWCC
NNC
The overview of geometric flow network (GFNet). Point clouds are first projected to range-view (RV) and bird's s-eeevview (BEV) using spherical and top-down projections, respectively. Then two branches with the proposed geometric flow module (GFM) handle RV/BEV to generate feature maps (H x W x C). Finally, grid sampling based on corresponding projection relationships is utilized to get the probability (N x C) for each point, and the fused prediction Fr is obtained by applying kpconv on the concatenation of RV RV/BEV. Note that the subplot bottom right corner illustrates how grid sampling works with dimension changing annotation, i.e., sampling F (F, or F5) with N C from M (M, or M,) with H x W C.
Figure 3:
Projection- Methods. Point clouds are first projected 2D images, e.g., rango- (RV) (Alonso
et al. 2020 Cortinhal al., 2020
Milioto et al. 2019 Wu et al.
2018
2019
Xu al., 2020) and convolutions. For example, as the backbone to process
bird's merd's-cyc-viww (BEV) (Zhang et al. 2020b), and then processed using
2D
++ngeNet++ (Milioto et al. 2019) adopts DarkNet (Redmon & Farhadi
2018
2020 standing on the shoulders RV images, and uses a KNN for post-processing SqueezeSegV3 (Xu et al.
Wu et al. (2018 2019), employs a spatially-adaptive Convolution (SAC)
to adopt different filters for dif-
ferent locations according to input RV images. SalsaNext (Cortinhal et al.
2020 introduces a new context
module which consists of a residual dilated convolution stack fuse receptive fields al various scales. the other hand, PolarNet (Zhang et al., (2200b) uses bosar-based birds-eye-viers (BEV) instead of the standard 2D Cartesian BEV projections to better model the imbalanced spatial distribution of point clouds.
Among project bon-based methods, applying multi-view projection can leverage rich complementary informa-
tion Alnaggar et al.) 2021 Chen et al.) 2020 Gerdzhev et al. 2021 Liong et al. 2020). while previous works
usually process RV BEV individually in sequence (Chen et al. 2020 Gerdzhev et al. 2021 or perform a
vanilla late fusion AInaggar et al. 2021 Liong et al.] 2020). For example, MVLidarNet (Chen et al., 2020 first obtains predictions from the RV image, which are then projected to BEV as initial foatures
learn al.) 2021) conducts in
representation by feature pyramid networks. Differently, TornadoNet (Gerdzheev reverse order by devising pillar-proecction--eemiii module (PPL) to extract features from BEV, and then placing these features into RV, modeled by encoder-decoded CNN. On the other side, MPF (Alnaggar ct al. 2021) utilizes two different models to scparately process RV and BEV, and then combines the pre- dicted softmax probabilities from two branches as final predictions. AMVNet (Liong et al., 2020) lakes a further step, i.e., after obtaining the separate predictions from RV and BEV, it adopts a point head (Qi let al.) 2017a) to refine the uncertain predictions, which are defined by the disagreements of two branches. Whereas, our GFNet enables geometric correspondence information to flow between RV RV/BEV at multi-levels during end-to-end learning, leading to more discriminative representation and better performances.
4

---

Published in Transactions on Machine Learning Research (09/2022)
3 Method
In this section, we first provide an overview of point cloud semantic segmentation and the proposed geometric flow network (GFNett). We then introduce projection-based point cloud segmentation using range-view (RV) and bird's-cyo-view (BEV) in detail. After that, describe the proposed geometrie flow module (GFM), including geometric alignment and attention fusion. Lastly, the end-to-end optimization of GFNet is depicted.
3.1 Overview
Given lidar point cloud with N 3D points P E RNx44 we then have the format. of each point (z. y, 2, remission), where (z, 2) is the cartesian coordinate of the point relative to the lidar sensor and remission indicates the intensity of returning laser beam. The goal of point cloud semantic segmentation is to assign all points in P with accurate semantic labels, i.e., Q E N For projection based point cloud semantic segmentation, we also need to transform the ground truth labels Q to the projected views during training, i.e., Q for RV and Qo for BEV.
The overall pipeline of GFNet is illustrated in Figure Specifically, a point cloud P first transformed range-view (RV) as I, and bird's sird''-eee-view (BEV) as I, using spherical and top-down projections, respectively. We then have two sub-network branches with encoder-decoder architectures to take RV RV/BEV images inpuls and generale dense predictions, which are referred to as the probability maps for each semantic class. The proposed geometric flow module (GFM) is incorporated into each layer of the decoder, bidirectionally propagating feature information according to the geometric correspondences across two views. After that, we obtain the classification probabilities of all points by applying a grid sampling on the dense probability maps, which based on the projection relationship between specific view and the original point cloud, illustrated in the bottom right corner of Figure 3 Inspired by Kochanov
al. (2020), we also introduce KPConv TTomas al. 2019 the top of the proposed GFNet to replace the KNN operation and capture the accurate neighbour information in learnable way. By doing this, the overall multi-view point cloud semantic segmentation pipeline can be trained in an end-to-end manner.
3.2 Multi-View Projection
For projection-based methods, point cloud P R-Nx4 needs to be transformed to image HWC first to leverage deep neural networks primarily developed for 2D visual recognition, where H and W indicate the spatial size of projected images and C' is the number of channels. Different projections are corresponding to different transformations, i.e., P Nx HWWx In this paper, we adopt two widely-used projected views for point cloud analysis, i.c., range-view (RV) and bird's-eye-view (BEV). As shown in Figure 3 we aim to learn effective aepresentations from two different views, RV and BEV, using the proposed two- branch networks with an encoder-decoder architecture in each branch. We deseribe the details of multi-view projection follows.
Range- View (RV). To learn effective representations from RV images, spherical projection is required to first project point cloud P to a 2D RV image (Milioto et al. 2019). Specifically, we first project a point (z, 2, from the cartesian space to the spherical space as follows:
1p Φ
arctan(y,2)
=
)
(1)
r
2 + 2
where vi o, and r indicate azimuthal angle, polar angle, and radial distance (ie., the range of each point), respectively. We then have the pixel coordinate of (I,y.2) 2) in the projected 2D range image as
(1- (Fup - -) H
(2)
5

---

Published in Transactions on Machine Learning Research (09/2022)
where (H, W) represent the spatial size of range image, and f = fup fdown is the vertical field-of-vies of the lidar sensor. For each projected pixel (u,z) (discretized from 0)), we take the (x,, V, 2, remission) as its feature, leading to a range image with the size of (H, W,5). In addition, an improved Mange-projecti method is proposed by (Triess et al.] (2020), which further unfolds the point clouds following the captured order by the lidar sensor, leading to smoother projected images and a higher valid projection rate]] Tf not
otherwise stated, we adopt this improved range projection (Triess et al.) 2020) all our experiments.
Bird's Eeess--y--iiie View (BEV). To learn effective representations from BEV images, top-down orthogonal pro- jection is employed to transform point cloud into BEV image (Chen et al., 2017c). Furthermore, the polar coordinate system is introduced to replace the cartesian system by Zhang et al. (2020b), which can be formulated as follows:
Vx2 + r" cosfarctan(y, polar(s, (1. v): x² + y² sin(arctan(y. x))]
where polar(.) is the coordinate transformation from cartesian system to polar system. Following Zhang let al. (2020b) we use nine features to describe each pixel vu (by discretizing (ū,ū) to [0, / I and 0, W 11) in BEV image, including three relative cylindrical distance, three cylindrical coordinates, two cartesian coordinates and one remission, which can be constructed as follows:
AAcliindrical(z, y, 2), cylindrical 2), I, V. remission)),
(4)
where cylindrical 2) = [polar(z (I. y), 2| represents the cylindrical coordinates, Acylindrical(z; y, 2) are the relative distances to the center of the BEV grid, and each BEV image thus has the shape of (H, W, 9).
3.3 Geometric Flow Module
Intuitively, RV and BEV contain different view information of the original point cloud through different projections, leading to different information loss on different classes. For example, RV is good at those tiny or vertically-extenndee objects such as motorcycle and person, while BEV is sensitive to those objects with large and discriminative spatial size on the x-y plane. To sufficiently investigate the complementary information from RV/BEV, we explore them from geometric perspective. Specifically, we devise a geometric flow module (GFM), which is based on the geometric correspondences between RV and BEV, to bidirectionally propagate the complementary information across different views. As illustrated in Figure 4 the first step is referred to as Geometric Alignment, which aligns the fcature of source view (RV or BEV) to the target view using their geometric transformation; then the second step is called Attention Fusion, which applies the self-ateention and the residual connection to combine the aligned feature representation with the original one. We describe the above-mentioned key steps of the proposed GFM module in detail as follows.
Geometric Alignment. The key idea lies in the geometric transformation matrices between two views, i.., TR-B (from RV to BEV) and TB-ı (from BEV to RV). To obtain these transformation matrices, we propose to utilize the original point cloud as an intermediary agent. Specifically, from Eq. and 2 we have the transformation from RV to the point cloud P as follows:
110.0
Do,
TR+P=
(5)
n1,-110
nH -1
where TR-P E ZH, (H,, W.) are the spatial size of 2D RV image, and ((me)) 0 i
<= W, 14.-1 1} is the (ni,)es point which projects on (i.j) coordinates. Note that if multiple points project to the same pixel, the point with smaller range is kept; If pixel is not projected by any points, then its nij is assigned as -. We then have the transformation from P to BEV image according to Eq.
T TP-B-
Un
uN 1]
T
(6)
T
Up
WN
(7)
60
UN
Please refer Appendix. for mare details.
6

---

Published in Transactions on Machine Learning Research (09/2022)
Cor
BN
Aliggmestt
ddittion
0g
Concatenation Multiplication
N-1
1
Xg
Yo
Ny--1
H6
X
Can
BNN
Figure 4: An overview of the proposed geometric flow module (GFM). It contains two main steps, i.e., geometric alignment and attention fusion, which first aligns the feature from the source view (RV or BEV) to the target view using their geometric correspondences, and then applies self-attention and residual con- nections to combine view-specific features with the flowed information. Note that H4 and 0, share the same architecture but not weights with μb and 0s respectively.
where TP-B E Nxx2 and {ux (ug, ux)l <= <= N 1} are the projected pixel coordinates of 2D BEV image, corresponding to the ke, point.
Algorithm Geometric Flow Module (BEV - RV)
RV feature map M, [H, Wr, Crl, BEV feature map M, [H, Wo, Col, TR-B [H,, W,, 2]. Output: Fused RV leature map fused [H,, Wr, C.].
Input:
Step 1: Geometric Alignment
Zero-initializing aligned feature Ma with the shape of [H,, W,, Ctl: foreach (i.j) E [1 W.] x [,] do
u [
Mo-,, (6 j. My(u,
Step 2: Attention Fusion
Concatenating M, and Ma along the channel dimension as Meoneat' [H,, Wr. Cr + Ct.
with the shape of Applying the self-attention module get Motten = μl ((Menooa)) [Hr, W,, Cr]
Employing residual connection Mussed = M, + Matten
Lastly, we calculate the transformed matrix TR-AB via TR-, and Tp-B. particular, for each pixel (i,j) in RV image, we first get its 3D point ni,j TR-Pli.j], then project n,, BEV image obtain the corresponding pixel Tp-B|nuJ Uniy Now, we obtain TR E HH,,
Uno.a
Ano,W,-
T TR-B=
(8)
U Ung.-110
Upn L,W,
Once obtaining TR+B, we can then align BEV features to RV features as follows: for each location (i,j) in RV image, the (u, u) coordinates in BEV image can be fetched via T TR-Bli, j], and then we fuse the feature in (u, v) to (i.j) get aligned feature Fo.r.. To align features from RV BEV, we can operate similar way with T T E xxxx2

---

Published in Transactions on Machine Learning Research (09/2022)
Attention Fusion. After the geometric feature aligmment, we employ attention fusion module obtain the fused feature by concatenating the aligned feature and the target feature, which is followed by two convolution operations pa-) and 0(-). They have simple architectures "Conv-BN-RELU" and "Conv-BN- Softmax" respectively, where the softmax function # is to map values to [0, 1] as attention weights. The atlention feature is finally combined with the larget feature using a residual connection. We demonstrate the overall process of fusing BEV to RV, including geometric alignment and attention fusion modules, The geometric flow from RV to BEV can be calculated similarly.
Algorithm !
3.4 Optimization
Given Qr as the labels for RV image I, and Qs for BEV image Ib which are projected from the original point cloud label Q we then have the 2D predictions M. for RV and M, for BEV, respectively. After that, obtain the 3D prodictions via grid sampling and KPConv, f.e., F, for RV and Fb for BEV. After fusion, get the final 3D predictions Fr. For simplicity and better illustration, we also highlight all predictions, To train the proposed GFNet, we first use the loss functions L2D and Lan for 2D and 3D predictions, respectively, as follows:
i.e., Mr, M, and Fr, Fb. Fr. in Figure 3
C20 = p- poLCoLLMM. Q+) to. CCL Qb),
(9)
and
CDD B- (F, Q) + CC.
(10)
where Lcg indicates the typical cross entropy loss function while LoL is the combination of the cross entropy loss and the Lovasz-Softmax loss (Berman et al. 2018) with weights 1 1. We then apply the cross entropy loss on the final 3D predictions Fr, that is, the overall loss function Liaat can be evaluated as:
Ltotal a (FF(Fr. Q) + LaD + Lan,
(11)
where X = [a, y, p. o] indicates the weight coefficient of different losses, and we investigate the influences of different loss terms Soc. 4.4
4 Experiments
In the soction, we first introduce the adopted SemanticKITTI (Behley et al. 2019 and nuScenes (Cacsar et al. 2020) datasets and the mean IoU and accuracy metric for point cloud segmentation. We then provide the implementation details of GFNet, including the network architectures and training settings. After that, perform extensive experiments to demonstrate the effectiveness of GFM and analyze the influences of different paper-parameters in GFNet. Lastly, we compare the proposed GFNet with recent state-of-the--e point conttppoojction-baae based methods to show our superiority.
4.1 Datasets and Evaluation Metrics
SemanticKITTI (Behley et al. 2019). derived from the KITTI Vision Benchmark (Geiger et al. 2012 provides dense point-wise annotations for semantic segmentation task. The dataset presents 19 challenging classes and contains 43551 lidar scans from 22 sequences collected with a Velodyne HDL-64E lidar, where each scan contains approximately 130k points. Following Behley al. (2019h; Milioto al. (2019) these sequences are divided into 3 sets, i.c., training set (00 to 10 except 08 with 19130 scans), validation set (08 with 4071 scans) and (esting (11 to 21 with 20351 scans). We perform extensive experiments on the validation set to analyze the proposed method, and also report performance on the test set by submitting the result to the official test server.
nuScenes (Caesar et al. 2020 is a large-scale autonomous driving datasel, containing 1000 driving scenes of 20 second length in Boston and Singapore. Specifically, all driving scenes are officially divided into training (850 scenes) and validation set (150 scenes). By merging similar classes and removing rare classes, point cloud semantic segmentation task uses 16 classes, including 10 foreground and 6 background classes. We use the official test server to report the final performance test set.
8

---

Published in Transactions on Machine Learning Research (09/2022)
Evaluation Metrics. Following Behley al. (2019), we use mean intersection-over--iiio (mloU) over all classes as the evaluation metric. Mathematically, ll, the mIoU can be defined as:
TP. mIoU - PC+
(12)
where Ta2 Fec, and FN: represent the numbers of true positive, false positive, and false negative predictions for the given class e, respectively, and C' is the number of classes. For comprehensive comparison, we also report the accuracy among all samples, which can be formulated as:
TP+TN Accuracy = TP+FP- FN +TN
(13)
4.2 Implementation Details
For SemanticKITTI we use two branches to learn representations from RV/BEV in an end-to-end trainable way, where each branch follows an encoder-decoder architecture with a ResNet-34 (He et al. 2016) as the backbone. The ASPP module (Chen et al. 2017b) is also used between the encoder and the decoder. The proposed geometric flow module (GFM) is incorporated into each upsampling layer. Note that the elements of TR-B, T fed into GFM are scaled linearly according to the current flowing feature resolution. For RV branch, point clouds are first projected to a range image with the resolution [64, 2048), which is sequentially upsampled bilinearly to [64 x 2S,, 2048 x S] where S is scale factor. During training, horizontal 1/4 random crop of RV image, i.e., [1288, 512S], is used as data augmentation. On the other hand, we adopt polar partition (Zhang et al. 2020bb for BEV, and use a polar grid size of [480, 360, 32] to cover a space of [radius: : (3m, 50m), 2 ( (-3m, 1.5m)] relative to the lidar sensor. The grid first goes through a mini PointNet (Qi et al. 2017a to obtain the maximum feature activations along the z axis, leading to a reduced resolution [480, 360 for BEV branch. We employ SGD optimizer with momentum 0.9 and the weight decay 1 We use the cosine learning rate schedule (Loshchilov & Hutter 2016) with warmup the first epoch to 0.1. The backbone network is initialized using the pretrained weights from ImageNet (Deng et al. 2009).
By default, we use X = [2.0, 2.0, 2.0, 1.0, 1.0] as the loss weight for Eq [1 We train the proposed GFNet for 150 epochs using the batch size 16 on four NVIDIA A100-SXM4-40GB GPUs with AMD EPYC 7742 64-Core Processor.
For nuScenes, we adopt Milioto et al. 2019 to project point clouds to a RV image with the resolution in our experiments. Besides, a polar grid size of [480, 360, 32] is used to cover relative space of [radius: (0m, 50m), =: : (-5m, 3m)] for BEV branch. We train the model for total 400 epoch with batch size 56 using 8 NVIDIA A100-SXM4-40GB GPUs under AMD EPYC 7742 64-Core Processor. We adopt cosine learning rate schedule (Loshchilov & Hutter 2016) with warmup at the first 10 epoch to 0.2. Other settings are kept the same with SemanticKITTI.
[32, 1024] which then upsampled bilinearly to [32 x 33S 1024 S] where
4.3 Effectiveness of GFM
In this part, we show the effectiveness of the proposed geometric flow module (GFM) well as its influences on each single branch. As shown in Figure 3 we denote the results from F, and F, as RV-Flow and BEV-Flow, respectively, in regard to the information flow between RV and BEV brought by GFM. The predictions from Fr (obtained by applying KPConv on the concatenation of F, and Fp) are actually our final results, termed as GFNet. Note that the above results are evaluated using X = [2, 2, 2, 1, 1] for Eq II In addition, we train also each single branch separately without GFM modules, i.e., using X 20,2, 1,0) and x= = [0, 0, 2, 0, 1] for RV-Single and BEV-Single, respectively.
We compare the performances of RV/BEV SB/BE-SSinge and BEV/BEV-Flow in Table Specifically, we find that both RV and BEV branches have been improved by a clear margin when incorporating with the proposed GFM module, e.g., 55.7% + 61.0% for BEV. Intuitively, RV is good at those vertically-extendee objects like motoreycle and person, while BEV sensitive to the classes with large and discriminative spatial size on the x-y plane. For example, RV-Single only achieves 32.4% on truck while BEV BEV-Single obtains 64.8%, which is also illustrated by the first row of Figure where RV predicts truck as mixture of truck, car
9

---

Published in Transactions on Machine Learning Research (09/2022)
truck dedde
lecyclist
rad parking sidewalk aherggrand
building faxe vegetation
trunk
taraain
pole traffiosign
GT
RV
BEV
GEN
Figure 5: Visualization of RV and BEV. The view with the cyan contour helps the one with red. By incorporating both RV and BEV, our GFNet makes more accurate predictions.
and other-vehicle, but BEV acts much well. This is partially because truck is more discriminative x-y planc (captured by BEV) than vertical direction (captured by RV) compared to car, other-wehicle. With the information flow from BEV to RV using GFM, RV-Flow significantly boosts the performance from 32.4% to 69.9%. A similar phenomenon can be observed in the second row of Figure 5 where BEV misclassifies biegelist as trunk, since both of them are vertically-extended and also very close to each other, while RV prodicts preciscly. With the help of RV, BEV-Flow dramatically improves the performance from 55.7% to 61.0%. When further applying KPConv on the concalenation of RV/BEV-Flou, the proposed GFNel achieves the best performance 63.0%. These results demonstrate that the proposed GFM can effectively propagate complementary information between RV and BEV to boost the performance of cach other, as well as the final performance.
Table 1: Quantitative comparisons in terms of mloU to demonstrate the effectiveness of GFM on the validation set of SemanticKITTI.
other vehicle
muotorcy Ellct
other- other-gromne ground
mootrryy
vegelation
bieyelke
truck
bicyclist
rood
purliing
sidewallk
baikiing
Jence
trank
tomein
pole
traffic-sin
mloU
Mothod
RV- RV-Single
93.7
48.7
57.7
32.4
40.5
69.2
79.9
0.0
96.9
53.4
83.9
0.1
89.2
59.0
87.8
66.1
75.3
64.0
46.2
60.1
Flow
93.8
15.0
58.8
69.9
31.6
63.6
73.8
0.0
95.6
52.9
83.6
0.3
90.3
62.1
88.0
61.3
73.8
63.2
47.1
61.1
BEV BEV-Sin Single
93.6
29.9
42.1
61.8
26.8
48.1
74.0
0.0
9L.
45.9
80.7
1.4
89.2
46.3
86.9
61.4
74.9
56.8
41.6
55.7
BEV BEV-Plow
93.7
43.7
61.2
74.0
31.0
61.6
80.6
0.0
95.3
53.1
82.8
0.2
90.8
61.4
88.0
63.1
75.6
58.9
43.1
61.0
GFNet
91.2
63.2
74.9
32.1
69.3
83.2
0.0
95.7
53.8
83.8
0.2
91.2
62.9
88.5
66.1
76.2
48.3
63.0
Table Ablation studies of attention GFM, loss weight coefficient À and scale factor on the Se- manticK[TTI val set.
(a) Attention in GFM
(b) À and under A 1,
=
attention
elg
B
7
S
mloU
mloU
sigmoid
softmax
A
3
61.7
62.0
b
A
4
A
3
61.8
62.9
A
A
A
A
A
3
62.4
63.0
V
V
v
A
A
3
63.0
V
V
V
A
4
2
61.7
V
V
V
A
A
4
63.2
10

---

Published in Transactions on Machine Learning Research (09/2022)
Table 3: Comparisons under mloU, Accuracy and Frame Per Second (FPS) on SemanticKITTI test set. Note that the results of methods with are obtained from RangeNet++ Miilioto et al.] 2019). From top to down, the methods are grouped into bant,, projection- based and multi-view fusion models.
Erafffc etggn
trunk
Srown
mndoU
PPS
Mlthhod
PourNNes
G 201RA
0.3
0.1
0.2
0.2
61.6
15.8
35.7
41.4
12.9
17.6
14.0
Nei4
53.7
0.2
0.9
0.9
72.0
18.7
41.8
62.5
16.9
15.8
30.0
20.1
0.1
TeagentCooy
10
ml. 2015
85.8
1.3
12.77
11.6
10.2
17.1
20.2
0.5
82.0
15.2
66..7
9.0
82.8
44.2
73.5
42.5
55.3
30.2
22.
35.9
0.3
ASNI
ar
87.9
25.
0
39.0
29.2
54.2
57.6
67.4
21.3
74.3
1.8
65.
13.9
81.1
52.2
70.6
S7.8
30. 11
36.8
RandLa
Net 111
0120
94.2
25.0
25.8
40.3
28.9
43.2
48.2
7.2
90.7
00.3
73.7
20.4
56.9
56.3
S1.4
61.3
00.8
49.2
47.7
53.9
15..5
22
KPCae
Thimst
21774
96.0
50.2
12.5
53.4
61.3
01.5
61.6
11.8
68.8
01.8
72.7
31.6
90.5
64.2
51.8
09.2
69.1
50.4
47.4
08.8
90.3
16.5
17.3
61.3
29.2
06.7
30.8
WWaet
81.8
17.9
20.1
25.1
65.8
07.6
70.7
41.1
35.8
60.1
20.2
36.3
19.7
91.4
35.4
25.7
38.3
39.8
91.8
65.0
75.2
67.4
58.6
65.
61.0
17.9
62:2
89.0
Nos Klaing 11 10 Kiz
93.8
40.3
30.1
22.9
26.5
63.2
40.2
6.6
00.8
61.7
74.4
21.7
50.0
61.5
81.0
65.6
67.8
51.8
57.5
64.3
$0.0
36
JD-MinNNTER PC Vi Si
2000
9335
12.3
42.1
28.5
29.4
67.8
86.1
14.3
91.6
61.2
74.5
25.4
50.4
60..5
52.8
00.8
06.7
18.0
16.6
55.8
850-7
78
ScuresceeVV3 Xa mie M320
92.5
08.7
36.5
20.5
33,0
40.6
46.2
20.1
91.7
63.4
74.8
26.4
59.0
20.4
82.0
68.7
65.4
49.5
58.9
55.9
89.5
6
SalssNext
Cortinaal
06.5
58.3
32.9
53.5
52.5
Allagsit 2043
93.4
38.3
26.1
45.1
46.1
90.6
02.3
74.5
68.5
30.7
58.7
69.1
55.5
CORNAIXIN (Cartray
91.2
05.7
10.0
03.0
60.1
31.9
89.7
06.3
74.5
28.7
91.3
65.6
07.0
71.5
58.0
03.1
90.7
AMVNit [Laung 16 mi 2020
96.2
69.9
64.2
48.8
65.7
73.0
65.7
11.0
90.1
71.0
75.8
32.4
92.4
60.1
85.6
71.7
69.6
62.7
67.2
65.3
91.3
GPNet (1015)
950
553
15.3
31.7
17.3
02.8
57.3
08.7
98.6
72.5
8608
31.2
943
73.9
85.2
71.1
09.3
61.8
08.00
05:4
92/4
10
4.4 Ablation Studies
In this subsection, we first explore the impacts of attention mechanism in GFM, the loss weights À defined in [ and the scale factor S introduced in Sec. 4.2 In the default setting, we use the softmax attention with x [2, 2, 2, 1, 1] and S 3.
As shown in Table 2a without attention mechanism (é.e., no 6(-) and @ in Figure J, the performance 62.0% is obviously inferior to the counterparts 62.9% and 63.0%, indicating that the attention operation helps to focus on the strengths instead of weaknesses of source view when fusing it into target view. If not otherwise stated, we use the softmax attention in our experiments. We evaluate the influences of X = [a. 3, 7: P. a] in [2.0, 2.0, 2.0, 1.0, 1.0] the configuration d.
Table [b) where A = 1, V = 2, e.g., we have x = [a. Y. Specifically, when comparing the configurations a b and c, we see that that additional supervisions on dense 2D and each branch RV /BEVVV 3D predictions further improve model performance. When comparing and d, a large weight on 3D prediction brings a better result. Therefore, if not otherwise stated, we adopt A == [2.0,2 2.0, 2.0, 1.0, 1.0] for remainiin experiments. The scale factor 5 in Sec. 4.2 indicates the resolution of RV image, e.g,, when S = 3, we have [1285, 5125] = [384, 1536) and [1285, 2048S = [383, 6144] for training and testing, respectively. When comparing and e in Table 2b we find that a higher resolution significantly improves model performance, from 61.7% to 63.0%, further enlarging S from 3 to 4 only brings a slightly better performance. For better speed-accuracy tradeoff, we use S 3 in the default setting.
4.5 Comparison with Recent State-of-the-Arts
SemanticKITTI. For fair comparison with recent methods, we follow the same setting in Behley et al. (2019): Kochanov et al. 2020), i.e., both training and validation splits are used for training when evaluating on the test server. As shown in Table 3 GFNet achieves the new state-of-ther thett--r---rttte performance 65.4% mloU, significantly surpassing point-based methods (e.g., 58.8% for KPConv (Thomas al. 2019h) and single view models (e.g., 59.5% for Next (Cortinhal et al., 2020)). For multi-view approaches (Alnaggar et al. 2021
Chen et al.) 2020: Gerdzhev et al.) 2021 Liong et al. 2020), GFNet outperforms recent methods Alnaggar et al. (2021); y Chen et al. (2020); Gerdzhev et al. (2021) by large margin. Comparing with AMVNet Li- ong et al. (2020), GFNet clearly outperforms it on the point-wisc accuracy, i.c., 92.4% us. 91.3%. The superior performance of GFNet shows the effectiveness of bidirectionally aligning and propagating geometric information between RV /BEV. Notably, AMVNee requires to train models for RV RV/BEV branch as well as their post-processing Point head separately, while GFNet end-to-end trainable. Additionally, since the acquisition frequency of the Velodyne HDL-64E LiDAR sensor (used by SemanticKITTI) is 10 Hz, GFNet can thus run in real-time, i..., 10 FPS.
11

---

Published in Transactions on Machine Learning Research (09/2022)
Table 4: Comparisons on nuScenes (the test set) under mloU and Frequency Weighted IoU (or FW loU).
const- const-veholk
motonnycele
pedestrian
iralii coIe
dri sir-farnnns
other- flat
stdewaik
vegotation
FW IoU
barrior
iraler
track
termin
b0s
Method
PolarNot Zhang al
2020b
72.2
16.8
77.0
86.5
51.1
64.8
54.1
09.7
63.4
96.6
67.1
77.7
72.
87.1
77
69.4
87.4
AM
Liong et al
2020
79.8
32.1
82.2
86.4
62.5
81.9
75.3
72.3
83.5
66.1
97.0
67.0
78.8
74.6
90.6
76.1
89.5
GPNet [amns]
81.1
76.0
00.2
80.7
75.3
71.8
82.5
97.8
67.0
76.2
88.9
90.4
nuScenes. To evaluate the generalizability of GFNet, also report the performance on the testset in Table g by submitting results to the test server. Similarly, GFNet achieves superior mloU performance 76.1%, which remarkably outperforms PolarNet (Zhang et al. 2020b) and tights AMVNet (Liong et al. (020)) However, our result 90.4% under Frequency Weighted IoU (FW IoU) beats 89.5% from AMVNet (Liong et al. 2020), which consistent with the accuracy comparison on SemanticcKITT also reveals that GFNet performs much better on frequent classes while somewhat struggles on those rare/smi classes. Despite the good performance of GFNet, it is also interesting to further improve GFNet by addressing rarc/small classes from the perspectives of data sampling/ angmentation and loss function.
5 Conclusion
In this paper, we propose a novel geometric flow network (GFNet) to learn effective view representations from RV and BEV. To enable propagating the complementary information across different views, we devise geometric flow module (GFM) to bidirectionally align and fuse different view representations via geometric correspondences. Additionally, by incorporating grid sampling and KPConv to avoid time-consuming and non-differenniabble post-processing, GFNet can be trained in end-to-end paradigm. Extensive experiments SemanticKITTI and nuScenes confirm the elfectiveness of GFM and demonstrate the new stale-of-the-at performance on projection- based point cloud semantic segmentation.
There are some limitations of the proposed method, since it builds upon two specific point cloud projection methods. Specifically, both RV and BEV may not applicable to indoor datasets such as S3DTS (Armeni et al. 2016). For example, in indoor scene of the bookcase, common objects such as table and chair are distinguishable and meaningful in the vertical direction, while the height. information is missing for BEV. cycle by the lidar sensor, which typically appears in outdoor scenarios such autonomous driving (Please also refer to Appendix. B for more details and figures). Additionally, we apply the proposed geometric flow module in each decoder layer (i.e., the upsampling layers), while popular point cloud object detection frameworks don't have such a decoder structure. Therefore, it is also non-trivial to directly apply the proposed method for object detection, which will be the subject future studies.
Also, RV image requires
6 Acknowledgement
This work is partially supported by ARC project FL-170100117.
References
Eren Erdal Aksoy, Saimir Baci, and Selcuk Cavdar. Salsanet: Fast road and vehicle segmentation in lidar point clouds for autonomous driving. In IEEE Intelligent Vehicles Symposium, pp. 926-932, 2020.
Yara Ali Alnaggar, Mohamed Afifi, Karim Amer, and Mohamed ElHelw. Multi projection fusson for real- time semantic segmentation of 3d lidar point clouds. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 1800-1809, 2021.
Inigo Alonso, Luis Riazuclo, Luis Montesano, and Ana Murillo. 3d-mininct: Learning a 2d represcntation from point clouds for fast and efficient 3d lidar semantic segmentation. IEEE Robotics and Automation Letters, 5(4):5432 5439, 2020.
12

---

Published in Transactions on Machine Learning Research (09/2022)
Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang, Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In TEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1534 1543, 2016.
Jens Behley, Martin Garbade, Andres Milioto, Jan Quenzel, Sven Behnke, Cyrill Stachniss, and Jurgen Gall. Semantickitti: A dataset for semantic scene understanding of lidar sequences. In IEEE/CVE International 2019.
Conference on Computer Vision (ICCV), pp.
Maxim Berman, Amal Rannen Triki, and Matthew B Blaschko. The lovász-softmax loss: A tractable surrogate for the optimization of the intersection-over-uiiins measure in neural networks. In IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 4413-4421, 2018.
Holger Caesar, Varun Bankiti, Alex H Lang. Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. In TEEE/CVE Conference on Computer Vision and Pattern Recognàtion (CVPR), pp- 11621-11631. 2020.
Ke Chen, Ryan Oldja, Nikolai Smolyanskiy, Stan Birchfield, Alexander Popov, David Wehr, Ibrahim Eden, and Joachim Pehserl. Mvlidarnet: Real-time multi-class scene understanding for autonomous driving using multiple views. In IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pp.
2288-2299 2294, 2020.
Liang-Chin Chich Chen, George Papandreou, Tasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 40(4):834-848, 2017a.
Liang-Chine Chich Chen, George Papandreou, Florian Schroff, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. XrXin preprint or arxiv: 706.05587, 2017b.
Xiaozhi Chen, Huimin Ma, Ji Wan, Bo Li, and Tian Xia. Multi-view 3d object detection network for autonomous driving. In IEEE/CVP Conference Computer Vision and Pattern Recognition (CVPR), 2017c
pp.
Ran Cheng, Ryan Razani, Ehsan Taghavi, Enxu Li, and Bingbing Liu. 2-s3net: Attentive feature fusion with adaptive feature selection for sparse semantic segmentation network. In IEEE/CVE Conference on Computer Vision and Pattern Recognétion (CVPR), pp. 12547-12556, 2021.
Tiago Cortinhal, George Tzelepis, and Eren Erdal Aksoy. Salsanext: Fast, uncortainty-aware semantic segmentation of lidar point clouds. In International Sympusium on Visual Compuling (ISVC), pp. 207 222, 2020.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image databasc. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248 255, 2009.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354-3361, 2012.
Martin Gerdzhev, Ryan Razani, Ehsan Taghavi, and Liu Binghing. Tornado-nct: multiview total variation semantic segmentation with diamond inception module. In IEEE International Conference on Robotics and Automation (ICRA), pp- 9543 9549, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 770 778, 2016.
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of largo-scale point clouds. In IEEE/CVF Con- ference on Computer Vision and Patlern Recognition (CVPR), pp. 11108-1117, 2020.
13

---

Published in Transactions on Machine Learning Research (09/2022)
Matterport Inc. Matterport 3d models of interior spaces. http://natterporttccom 2015. Accessed: 2015- 06-01.
Deyvid Kochanov, Fatemeh Karimi Nejadasl, and Olaf Booij. Kprnet: Improving projection-based lidar semantic segmentation. arXiv preprint or arXiv:2007. 12668, 2020.
Xuyou Li, Shitong Du, Guangchun Li, and Haoyu Li. Integrate point-cloud segmentation with 3d lidar scan-matching for mobile robot localization and mapping. Sensors, 20(1):237, 2019.
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolution on x-transformed points. Advances in Neural Information Processing Systems (NeurIPS), 31:820-833 2018.
Ying Li, Lingfei Ma, Zilong Zhong, Fei Liu, Michael A Chapman, Dongpu Cao, and Jonathan Li. Deep lcarning for lidar point clouds in autonomous driving: A review. IEEE Transactions on Neural Networks and Learning Systems (TNNLS), 32(8):3412 -432, 2020.
Venice Erin Liong, Thi Ngoc Tho Nguyen, Sergi Widjaja, Dhananjai Sharma, and Zhuang Jie Chong, Amvnet: Assertion-based multi-view fusion network for lidar semantic segmentation. arXiv preprinl arXiv:2012.0490 04994, 2020.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmenta- tion. In IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pp. 3431 8431-3440, 2015.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXio preprint arXiv:1608.0300 03983, 2016.
Andres Milioto, Ignacio Vizzo, Jens Behley, and Cyrill Stachniss. Rangenet+++: Fast and accurate lidar semantic segmentation. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 4213 -2200, 2019.
Charles R. Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognation (CVPR), pp. 652 660, 2017a.
Charles R Qi, Li Yi, Hao Su, and Lconidas J Guibas. +ointnct+ Deep hicrarchical feature learning on point sets in a metric space. arXiv preprinl 1706 02413, 2017b.
Joseph Redmon and Ali Farhadi. Yolov3: An incremental improvement. arXiv preprint arXiv:1804.0277 2018.
Haotian Tang, Zhijian Liu, Shengyu Zhao, Yujun Lin, Ji Lin, Hanrui Wang, and Song Han. Searching efficient 3d architectures with sparse point-voxel convolution. In Europeun Conference on Compuler Vision (ECCV), pp. 685-702, 702, 2020.
Liyao Tang, Yibing Zhan, Zhe Chen, Baosheng Yu, and Dacheng Tao. Contrastive boundary learning for point cloud segmentation. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8499, 2022.
Maxim Tatarchenko, Jaesik Park, Vladlen Koltun, and Qian-Yi Zhou. Tangent convolutions for dense prediction in 3d. In IEEE/CVF Conference Computer Vision and Pottern Recognation (CVPR), pp. 3887-380 3896, 2018
Hugues Thomas, Charles R Qi, Lean-Emmaant Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In IEEE/CVF Inter- national Conference on Computer Vision (ICCV), pp. 6411-6420, 2019.
Larissa T Tricss, David Peter, Christoph B Rist, and J Marius Zöllner. Scan-bascd semantic segmentation of lidar point clouds: An experimental study. In IEEE Intelligent Vehicles Symposimm, pp. 1116-1121, 2020.
14

---

Published in Transactions on Machine Learning Research (09/2022)
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions On Graphics (TOG), 38(5):1-12, 2019.
Bichen Wu, Alvin Wan, Xiangyu Yue, and Kurt Keutzer. Squeezeseg: Convolutional neural nets with recurrent erf for real-time road-object segmentation from 3d lidar point cloud. In IEEE International Conference Robotics and Automation (ICRA), pp. 1887-1893, 2018.
Bichen Wu, Xuanyu Zhou, Sicheng Zhao, Xiangyu Yue, and Kurt Keutzer. Squeezesegv2: Improved model structure and unsupervised domain adaptation for road-object segmentation from lidar point cloud. In 2019.
IEEE International Conference on Robolics and Aulomalion (ICRA), pp-
Chenfeng Xu, Bichen Wu, Zining Wang, Wei Zhan, Peter Vajda, Kurt Keutzer, and Masayoshi Tomizuka. Squeezesegv3: Spatially-daaptive convolution for efficient point-cloud segmentation. In European Confer- ence on Computer Vision (ECCV), pp. 1-19, 2020.
Jianyun Xu, Ruixiang Zhang, Jian Dou, Yushi Zhu, Jie Sun, and Shiliang Pu. Rpvnet: A deep and effi- cient range-point-voeel fusion network for lidar point cloud segmentation. In IEEE/CVF International Conference on Computer Vision (ICCV), pp. 16024-16033, 2021.
Xu Yan, Jiantao Gao, Jie Li, Ruimao Zhang, Zhen Li, Rui Huang, and Shuguang Cui. Spanse single sweep lidar point cloud segmentation via learning contextual shape priors from scene completion. arXic preprint arXiv:2012.0377 2020a.
Xu Yan, Chaoda Zheng, Zhen Li, Sheng Wang, and Shuguang Cui. Pointasnl: Robust point clouds processing using nonlocal neural networks with adaptive sampling. In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5589-5598, 2020b.
Xu Yan, Jiantao Gao, Chaoda Zheng, Chao Zheng, Ruimao Zhang, Shuguang Cui, and Zhen Li. 2dpass: 2d priors assisted semantic segmentation lidar point clouds. In European Conference Computer Vision (ECCV), 2022.
Lei Yang, Yanhong Läu, Jinzhu Peng, and Zize Liang. A novel system for off-line 3d seam extraction and path planning based on point cloud segmentation for are welding robot. Robotics and Computer-Integraeed Manufacluring, 64:101920 2020.
Maoshong Ye, Rui Wan, Shuangjic Xu, Tongyi Cao, and Qifeng Chen. Drinct Efficient voxcl-as-point point cloud segmentation. Xiiie preprint or arx:v::111
2021.
Feihu Zhang, Jin Fang, Benjamin Wah, and Philip Torr. Deep fusionnet for point cloud semantic segmenta- tion. In European Conference on Computer Vision (ECCV), pp. 644 163,666, 2020a.
Yang Zhang, Zixiang Zhou, Philip David, Xiangyu Yue, Zerong Xi, Boqing Gong, and Hassan Foroosh. Polar- net: An improved grid representation for online lidar point clouds semantic segmentation. In IEEE/CVF Conference Computer Vision and Pattern Recognation (CVPR), pp. 9601 9610, 2020b.
Hengshuang Zhao, Jianping Shi, Xiaojuan Qi, Xiaogang Wang, and Jiaya Jia. Pyramid scene parsing network. In IEEE/CVF Conference on Compuler Vision and Pallern Recognition (CrioonnCC (CVPR), pp. 2881 2890, 2017.
Xinge Zhu, Hui Zhou, Tai Wang, Fangzhou Hong, Yuexin Ma, Wei Li, Hongsheng Li, and Dahua Lin. Cylindrical and asymmetrical 3d convolution networks for lidar segmentation. In IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 9939-9948 2021.
15

---

Published in Transactions on Machine Learning Research (09/2022)
A RV Projection
(a) Original RV
(b) Improved RV
Figure 6: Range images from original RV Milioto et al. 2019 and improved RV Triess et al. 2020 As
we can see, Triess et al. 2020)) obtains smoother projected image than Milioto et al. (2019).
We project 3D point cloud P to a 2D RV image with the size of (H, W): due to the 2D-to-3D ambiguity, there are pixels that are not projected by any points, while multiple points might be projected to the same pixel. We define the valid projection rate as the ratio of valid pixels (i.e., projected by as least one point) comparing to total pixels HW. Specifically, more valid pixels usually result in smoother projected image, i.e., the higher valid projection rale, the better. We compare two different projection methods (Miliolo
Table 5: Valid projection rate (%) when using two
different RV projections (Milioto et al. 2019 Triess
et al. 2020 to generate images of 64 2048 size.
Method
Train
Val
Original RV Milloto et al.
2019
72.47
72.12
Improved RV (Triess et al.
2020
83.69
83.51
et al. 2019 Triess et al. 2020) in terms of valid projection rate in 5 As we can observe, Triess et al. (2020)
significantly outperforms Milioto et al. (2019) by over 11% in both train and val set. In addition, we visualize
the RV images oblained by Milioto el al. ( Triess et al. (2020) separately in Figure 6 Obviously, the
RV image generated by Triess et al. (2020) is clearly smoother than Milioto et al. (2019). Therefore, we use
Triess et al. 2020) in all experiments if not states otherwise.
B RV under Indoor and Outdoor Scenes
In this section, we first described the RV projection (i.e., spher- ical projection) in detail. We then show the difference between the point clouds collected in outdoor and indoor scenes. Lastly, we discuss why RV projection is not suitable for indoor scenes.
As shown in Figure 7 given a 3D point we first obtain the corresponding t ø for spherical projection. We then normalize Φ, s to [o, 1] and map them to 2D RV image with size [H, W] ac- cording to Fq. 2 However, point clouds in outdoor and indoor scenes are usually collected in different ways. For example, SemanticKITTI is collected via a Velodyne HDL-64E lidar the top of the driving car, which launches lasers to all-around (360%) horizontal directions and a certain degree [fdowns fupl vertical directions. When applied RV projection, a meaning- ful projected cylindrical image can be obtained (please refer to Figure D_ But for indoor dataset like S3DIS (Armeni et al. 2016), scans the entire room in any directions with a Matter- port [Inc.] 2015) scanner to generate point clouds. In addition, those indoor objects
p
1
Figure 7: Range-view projection (i.e., spherical projection). p is 3D point, 3/ are the azimuthal angle, polar angle of p.
much more dense than outdoor ob- jects. If we still want to use RV projection, it will lcad to severe distort image. We have also made some altempts using RV projection for S3DIS, but obtained mean- ingless images as in Figure 9 That is also the reason why existing projection-based methods (Milioto et al. 2019 Wu et al., 2018 Cortinhal et al. 2020) only employ RV projection in outdoor lidar-collected point clouds. As for indoor datasets like S3DIS, the mainstream methods (Qi et al. 2017a] Thomas et al. 2019) usually take raw points as input directly given that their size is much smaller than outdoor scenes.
16

---

Published in Transactions on Machine Learning Research (09/2022)
Jup
down
Figure 8: A simple diagram of the working mechanism when the lidar sensor collects point clouds. The lidar launches lasers to all-around (3609) horizontal directions and certain degree [fdown: fupl vertical directions. Note that the vertical field of view f = Sup Sdown where Sdown is negative.
Figure 9: Three visualizations of samples from S3DIS Armeni el. al. 2016 using RV projection.
C
Visualization
Figure 10 illustrates comparisons weew GFNet and ground truth on complex scenes, revealing the excel- lent performance of GFNet. We also provide a GIF image, i.e., figs/vis.ght at https://github. com/ haibo-qiu/GFNeet for more visualizations.
17

---

Published in Transactions on Machine Learning Research (09/2022)
car
bicycle
motorcycle
truck
other-vehicle
person
bicyclist
motorcyclist
road
parking
sidewalk
other-ground
building
fence
vegetation
trunk
terrain
pole
traffic-sign
(a) Preds
(b) GT
Figure 10: Visualization of the predictions from our GFNet comparing to GT.
18


Reviews:
[{"review_id": "q6uvJtQLTL", "reviewer": "Reviewer_bVo2", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The article is well written, easy to follow, and has adequate references. The literature review is comprehensive, well organized, and up-to-date. I may suggest, however, including some more recent references. \n\nFigures 1 and 2 help understand the method and Figure 3 clearly shows the architecture. GFNet is novel, simple, and end-to-end differentiable. The notation seems correct and adequate. \n\nThe experiments are conducted in relevant datasets and evaluated using standard metrics. The results outperform the state-of-the-art in all the cases. However, the results are not much better than these of AMVNet. Nonetheless, GFNet is easier to train than AMVNet. The ablation studies are complete.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "GFNet (geometric flow network) addresses the critical problem of 3D Semantic segmentation on point clouds.  GFNet explores the geometric correspondence between different views in an align-before-fuse manner, in contrast with traditional methods that use late fusion.  This new method achieves state-of-the-art compared to projection methods.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "I miss a section explaining the computational needs of the method for training and inference. It says that it works at 10FPS but I didn't see the computation requirements for that. Moreover, compare it with the speed of AMVNet which has almost the same mIOU.\n\nI miss an analysis of the weak points of GFNet and an agenda of future work on how to improve it.\n\nI miss some more recent references. ", "broader_impact_concerns": "The article has not had a broader impact statement. This method can be used for autonomous driving or can also be used for military purposes. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "itOKA9bFng", "reviewer": "Reviewer_A5Ne", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. 3D point cloud segmentation has indeed always been a fundamental task for point cloud related tasks. In this paper, it is feasible to use the projection of range view and bird's-eye view to complement each other, but the point clouds obtained from these two perspectives seem to have overlapping gaps, so how does this paper consider this problem? On the other hand, the translation from one perspective to another in this paper goes through three steps, and there seems to be a large loss of information in between.\n\n2. The point cloud segmentation task actually serves the subsequent downstream tasks in more cases. The method in this paper seems to have achieved a certain gain, so how much does this gain help the downstream tasks? For example, the task of 3D object detection.\n\n3. The methods proposed in this paper are all outdoor point cloud segmentation, so is it applicable to point cloud segmentation for indoor objects? If applicable, how did it perform? If not applicable, please explain why?\n\n4. The experimental results in this paper are performed on the KITTI and nuScenes datasets, so what is the performance on the larger dataset Waymo? This dataset seems to be more suitable for evaluating the tasks related to this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper explores the geometric correspondence between different views in 3D point clouds. Experiments on  KITTI and nuScenes datasets show promising results.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "refer to Strengths And Weaknesses.", "broader_impact_concerns": "No Broader Impact Concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R2GFxVHGx3", "reviewer": "Reviewer_m9mP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n(+) The paper follows an interesting line of work fusing multiple representations of 3D pointclouds for scene understanding. Most interestingly, these representations are projected from the input pcl as pre-processing and do not stem directly from some input sensor, which raises an important gap in our understanding of pcl processing where operating directly on the input isn't as capable. \n(+) Performance improvement over baselines is impressive. \n(+) Geometric Flow Module is well explained and seems easily reproducible. \n\nWeaknesses:\n(-) The two conv branches are named \"symmetric branches\" and I wonder what symmetry is used here? Are the conv weights shared? What is done with respect to the difference in input images size? it seems both branches use ResNet34 so i'd assume the difference in image sizes persists throughout the network? maybe indicate that in the figure?\n(-) Further, I'm missing an explanation of why should mu and theta be shared between the two branches? These are convolution layers that operate on very different input images.\n(-) It seems BEV is only better than RV in the truck category and anywhere else performs worse, and often much worse. I therefore wonder whether BEV really contributes or would it have sufficed to augment the 3D network (kpconv) with just the RV branch? To my understanding, the reported ablation does not evaluate this since when the BEV branch is deactivated (lambda =  [0, 2, 0, 1, 0]) also alpha == 0 so GFNet isn't active. More generally, this may indicate that these two representations are far from optimal, and potentially other projections of the input pcl are better. It would be interesting to have at least a discussion about that. Also, given that these projections are produced artificially by the authors as preprocessing and not naturally emerge from a sensor, it's worth discussing why current network architectures cannot reproduce the same performance. \n(-) The paper only tackles segmentation. Given how e.g. MV3D tested different fusion approaches, it will be interesting to adopt the geometric flow and compare it on detection tasks. \n(-) The paper only tackles outdoor scenes: indoor scenes have many details along the height axes, therefore a common criticism against BEV is that it may lose these important details necessary for indoor tasks. Here, complementary features coming from RV and finally from 3D KPConv therefore I think it's important to know whether these projections are also useful for indoor scenes. In particular, KPConv was evaluated on Scannet so seeing whether projected views can assist its performance would be insightful. \n(-) input features seem quite simplistic. For example, MV3D computed density and height maps for the BEV input features. Can the authors try this here as well?\n(-) Adding image modality seems natural and would further strengthen the work\n \n\nMinor:\n(-) If Phi ranges from f_down to f_up should v tilde be: (phi - f_down) / (f_up - f_down) * H ?\n(-) In Eq (3) polar is defined for (x,y) but in Eq (4) it is used with x,y,z input. How should the 3D input be understood?\n(-) Would be useful to mention why after transforming the cartesian coordinates to polar, both are concatenated. \n(-) in Attention Fusion computation, is Softmax computed globally on the entire image? \n(-) Note that BEV and RV are indicated by subscripts throughout the paper notation (e.g. Qr and Qb) but suddenly in the prediction the notation becomes Rc and Bc. I suggest keeping consistency with the rest of the work. \n(-) Further, under the current notation I believe the second loss term in Eq (9) should compare Bc with Qb and not Rb.\n(-) grid sampling is not explained: is this mapping from HxW to original points N? \n(-) bottom right illustration in figure 3 is confusing -- it's not clear where it fits in the large module. Seems to just repeat the output operations already listed? Also, the comma separated Rc and Bc notation is not explained. It should be clear they are not concatenated, and each has its own different H and W sizes and not share the same size. \n(-) L_CL is a combination of losses. I assume weighted? What are the weights?\n(-) how are the projected annotations Q_r and Q_b computed? what happens in projected view pixels that have points mapped to them with different classes? \n(-) what is the ASPP module mentioned?\n(-) what is meant by linear scaling of the features  \"according to the resolution\"? \n\n\nTypos:\n(-) In section 3.2: \"we aims\"\n(-) not a typo but \"generate probabilistic maps\" sounds vague. Do the authors mean semantic class probability here? maybe spell it out more explicitly. \n(-) kpconv in conclusion (fix capitalization)", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "A method for 3D pointcloud semantic segmentation is proposed. The core idea is to operate at two projected representations of the input 3D pcl: BEV and RV. At each decoder layer the two branches share information through a simple geometric alignment module that pushes features from each branch to the corresponding place on the other branch. Finally, both branches are combined and jointly processed in 3D to get the final prediction. Experiments show good overall performance with consistent improvement over using only BEV or RV or 3D. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "The conceptual and technical contribution of the work is rather small. Yet, performance is impressive and therefore opens up interesting questions. Aside for addressing all the clarifications asked under \"Strengths And Weaknesses\" i've also listed several ideas for improving the manuscript like adding tasks, testing different projections, discussing limitations of direct 3D approaches etc. ", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Two reviewers recommend acceptance (m9mP: Accept; bVo2: Leaning Accept) and one rejection (A5Ne: Leaning Reject). While the authors' responses addressed most of the initial reviewers' concerns, some questions raised by A5Ne remained open, such as the impact on downstream tasks (e.g., 3D object detection), the results on the larger Waymo dataset, and the applicability to indoor scenes. The Action Editor nonetheless believes that there is sufficient material for acceptance, as acknowledged by the other two reviewers. However, the authors are encouraged to incorporate their answers regarding 3D object detection and indoor scenes as limitations of the approach.
Decision: Accept (poster)