Semantic segmentation of sparse irregular point clouds for leaf/wood discrimination
Authors: Yuchen BAI
Jean-Baptiste Durand
Grégoire Laurent Vincent
Florence Forbes
Venue: NeurIPS 2023 poster (2023)
Keywords: UAV
Deep Learning
Semantic Segmentation
Lidar
Class Imbalance
Point Cloud
TL;DR: Semantic segmentation of UAV LiDAR point clouds for forests

Abstract: Lidar (Light Detection and Ranging) has become an essential part of the remote sensing toolbox used for biosphere monitoring. In particular, Lidar provides the opportunity to map forest leaf area with unprecedented accuracy, while leaf area has remained an important source of uncertainty affecting models of gas exchanges between the vegetation and the atmosphere. Unmanned Aerial Vehicles (UAV) are easy to mobilize and therefore allow frequent revisits to track the response of vegetation to climate change. However, miniature sensors embarked on UAVs usually provide point clouds of limited density, which are further affected by a strong decrease in density from top to bottom of the canopy due to progressively stronger occlusion. In such a context, discriminating leaf points from wood points presents a significant challenge due in particular to strong class imbalance and spatially irregular sampling intensity. Here we introduce a neural network model based on the Pointnet ++ architecture which makes use of point geometry only (excluding any spectral information). To cope with local data sparsity, we propose an innovative sampling scheme which strives to preserve local important geometric information. We also propose a loss function adapted to the severe class imbalance. We show that our model outperforms state-of-the-art alternatives on UAV point clouds. We discuss future possible improvements, particularly regarding much denser point clouds acquired from below the canopy.

Semantic segmentation of sparse irregular point clouds for leaf/wood discrimination
Yuchen Bai' Jean-Baptiste Durand Grégoire Vincent2 Florence Forbes'
Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, IJK, Grenoble, France
2 AMAP, Univ. Montpellier, CIRAD, CNRS, INRAE, IRD, Montpellier, France
{yuchen. florrecee.... (oorbes]@inria.... jean-baptiste dtw.rappiiii.duuuuddieieaadddddd durand@cirad. gregoire vincent@ird..f
Abstract
LiDAR (Light Detection And Ranging) has become an essential part of the remote sensing toolbox used for biosphere monitoring. In particular, LiDAR provides the opportunity to map forest leaf area with unprecedented accuracy. while leaf area has remained an important source of uncertainty affecting models of gas exchanges between the vegetation and the atmosphere. Unmanned Aerial Vehicles (UAV) are easy to mobilize and therefore allow frequent revisits, so as to track the response vegetation to climate change. However, miniature sensors embarked on UAVs usually provide point clouds of limited density, which are further affected by a strong decrease in density from top to bottom the canopy due to progressively stronger occlusion. In such context, discriminating leaf points from wood points presents significant challenge due particular to strong class imbalance and spatially irregular sampling intensity. Here introduce neural network model based on the Pointnet ++ architecture which makes use of point geometry only (excluding any spectral information). To cope with local data sparsity, we propose an innovative sampling scheme which strives to preserve local important geometric information. We also propose a loss function adapted the severe class imbalance. We show that our model outperforms state-of-the-art alternatives on UAV point clouds. We discuss future possible improvements, particularly regarding much denser point clouds acquired from below the canopy.
Introduction
In the past decades, LiDAR technology has been frequently used acquire massive 3D data in the field of forest inventory (Vincent et al. [] Ullrich & Pfennigbauer (2].. The acquisition of point cloud data by employing LiDAR technology is referred to as laser scanning. The collected point cloud data provides rich details on canopy structure, allowing us calculate a key variable, leaf area, which controls water efflux and carbon influx. Monitoring leaf area should help in better understanding processes underlying flux seasonality in tropical forests, and is expected to enhance the precision of climate models for predicting the effects of global warming. There are various types of vehicles for data collection, with groend-based equipment and aircraft being the most commonly employed. The former operates a bottom-up scanning called terrestrial laser scanning (TL.S), providing highly detailed and accurate 3D data. Scans often acquired in grid pattern every 10 and co-registered into a single point cloud. However, TLS requires human intervention within the forest, which is laborious and limits its extensive implementation. Conversely, airborne laser scanning (ALS) much faster and can cover much larger areas. Nonetheless, the achieved point density is typically two
"Correponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

orders of magnitude smaller due to the combined effect of high flight altitude and fast movement of the sensor. Additionally, occlusions caused by the upper tree canopy make it more difficult observe the understory vegetation.
In recent years, the development of drone technology and the decreasing cost have led UAV laser scanning (ULS) becoming one favored option (Brede et al. [3). It does not require in-situ cover a few hectares. The acquired data is much denser than ALS (see Figure [(a] and Figure [[b), which provides us with more comprehensive spatial information. Increasing the flight line overlap results in multiple angular sampling, higher point density and mitigates occlusions. Although the data density is still relatively low, compared with TLS, ULS can provide previously unseen overstory details due to the top-down view and overlap flight line. Furthermore, ULS is considered to be more suitable for conducting long- teggtermmnn monitoring of forests than TL.S, as allows predefined flight plans with minimal operator involvement.
intervention and each flight can be programmed
Consequently, leaf-wood semantic segmentation in ULS data is required to accurately monitor foliage density variation over space and time. Changes in forest foliage density are indicative of forest functioning and their tracking may have multiple applications for carbon sequestration prediction, forest disease monitoring and harvest planning. Fulfilling these requirements necessi- tates the development of a robust algorithm that capable to classify leaf and wood in forest environments. While numerous methods have demonstrated effective results on TLS data (see Section 2). these methods cannot be applied di- rectly to ULS, due in particular to the class im- balance issue: leaf points account for about 95% of the data. Another problem is that many meth- ods rely on the extra information provided by LiDAR devices, such as intensity. In the context of forest monitoring, intensity is not reliable due to frequent pulse fragmentation and variability in natural surface reflectivity (see Vincent et al. (5). Furthermore, the reflectivity of the vege- tation is itself affected by diurnal or seasonal changes in physical conditions, such as water content or leaf orientation (Brede et al. [3]. Therefore, methods relying on intensity infor- mation (Wu al. (6)) may exhibit substantial variations in performance across different locations and even within the same location for different acquisition batches. To address this issue, certain methods (LeWos proposed by Wang et al. [4; Morel et al. (7) have good results while exclusively utilizing the spatial coordinates of LiDAR data.
(a) ALS
(b) ULS
(c) TLS
Figure 1: Point clouds produced by three scanning
modes on the same area (20m x 20m x 42m), il- lustrate how much the visibility of the understory differs. The colors in the figure correspond to dif- ferent labels assigned to the points, where red and green indicate leaves and wood, respectively. Blue points are unprocessed, so labeled as unknown. TLS data were initially subjected to semantic seg- mentation using LeWos 4 and subsequently man- ually corrected. Next, the K-nearest neighbors (KNN) algorithm was used to assign labels to ALS and ULS data based the majority label among their five nearest neighbors in TLS data.
Inspired by the existing methods, we propose a novel end-to-end approach SOUL (Semantic segmen- tation On ULs) based on PointNet++ proposed by Qi et al. [8] to perform semantic segmentation on ULS data. SOUL uses only point coordinates as input, aiming lected in forests from various locations worldwide and with sensors operating at different wavelengths. The foremost concern be tackled is the acquisition of labeled ULS data. Since no such data set existed up to now, we gathered a ULS data set comprising 282 trees labeled as shown in Figure 4 This was achieved through semi-automatic segmentation of a coincident TLS point cloud and wood/leaf' label transfer to ULS point cloud. Secondly, the complex nature of tropical forests necessitates the adoption of a data preppartiiioning scheme. While certain methods (Krisanski et al. 19; Wu et al. (00) employ coarse voxels with overlap, such an approach leads to a fragmented representation and incomplete preservation of the underlying geometric information. The heterogeneous distribution of points within each voxel, including points from different trees and clusters at voxel boundaries, poses difficulties for data standardization. We introduce a novel data preprocessing methodology named geodesic voxelization decomposition (GVD), which leverages geodesic distance as metric for partitioning the forest data into components and uses the topological features, like Marinsic-cxtrinsie
be applicable to point clouds col-
2

---

ratio (IER) (He et al. [1;; Liu et al. [12), to preserve the underlying geometric features at component level (see Section D. The last issue concerns the class imbalance problem during the training stage. To address this issue, we developed novel loss function named the rebalanced loss, which yielded improved performance compared with the focal loss (Lin et al. [3] for our specifie task. This enhancement resulted in 23% increase in the ability to recognize wood points, see Table
The contribution of our work three-fold. First, SOUL is the first approach developed to tackle the challenge of semantic segmentation on tropical forest ULS point clouds. SOUL demonstrates better wood point classification in complex tropical forest environments while exclusively utilizing point coordinates as input. Experiments show that SOUL exhibits promising generalization capabilitiess achieving good performance even on data from other LiDAR devices, with a particular emphasis on overstory. Secondly. we propose a novel data preprocessing method, GVD, used to pre- pre-partition data and address the difficult challenge tropical forest environments. Third, we mitigate the issue of imbalanced classes by proposing new loss function, referred to as rebalanced loss function, which easy to use and can work as plug-and-play for various deep learning architectures. The data set (Bai et al. (14)) used in the article is already available in open access at https://emmddooor///coo/8888885 and our code is available at https://github.c com/Na
training neural networks from sparse point clouds in
2
Related Work
Classical methods. Classical methods are prevalent for processing point cloud data in forest environments, especially for semantic segmentation of TLS data, where they have achieved high levels of efficacy and applicability practice. One such method, LeWos (Wang et 4 uses geometric features and clustering algorithms, while an alternative method proposed by the first author of LeWos (Wang [15) employs superpoint graphs (Landrieu
Simonovsky [16].. Moorthy et
al. [17 use random forests (RF), and Zheng et al. [18 explore Gaussian mixture models. Other
approaches, such as those proposed by Lalonde al. [19,, Itakura et al. 20), Vicari et al. [21, and
Wan et al. [22], have also demonstrated effective semantic segmentation of TLS data.
Deep Learning methods. Pioneering works like FSCT proposed by Krisanski et al. 2 and the method proposed by Morel et al. 7 have shown promising results by using PointNet++ [ Shen et
al. (23] use PointCNN (Li et al. (24] the backbone model to solve the problem of segmenting
tree point clouds in planted trees. Wu et al. [6] propose FWCNN, which incorporates intensity information and provides three new geometric features. Windrim & Bryson (10) propose method that combines Faster R-CNN (25) and Pointnet [26). which utilizes the bird's eye view technique for single tree extraction and performs semantic segmentation individual trees. In general, deep neural network-basee approaches have demonstrated better performance in semantic segmentation of TLS point clouds compared classical methods.
(a) Raw Lidar data
(b) Geometric feature computation
Components proposed by GVD
(d) Training PointNet++
(e) Prediction
Figure 2: Overview of SOUL.. (a) We use only the coordinates of raw LiDAR data as input. (b) Four geometric features linearity, sphericity, verticality, and PCA1 calculated at three scales using cigenvalues, then standardized. (c) GVD proposes partitioned components and performs data normal- ization within these components. (d) Training deep neural network. (e) Finally, output are point-wise predictions.
3

---

3 SOUL: Semantic segmentation on ULS
SOUL is based on PointNet++ Multi-See Selti--caae Grouping (MSG) g with some adaptations. The selection of PointNet++ is not only because of its demonstrated performance in similar tasks (Krisanski et al. 19] Morel et al. [7]; Windrim & Bryson[[U] [O]), but also because of the lower GPU requirements (Choe al. (27] compared with transformer-base
models developed recent years, like the method proposed by Zhao et al. [28]. The main idea of SOUL lies in leveraging geometric approach to extract preliminary features from the raw point cloud, these features are then combined with normalized coordinates into a deep neural network to obtain more abstract features in some high dimensional space [26]. We will introduce our method detail as follows.
3.1 Geometric feature computation
At this stage, we introduce four point-wise features: linearity, sphericity, verticality, and PCAL, which are computed at multiple scales of 0.3 m, 0.6 m, and 0.9 m this task. calculate these features, we need to construct the covariance matrix Ep: for each point p4, compute its eigenvalues \1. > 2 > As,5 0 and the corresponding normalized eigenvectors 61,s 62 E3 The local neighbors Np, of pa are given by:
No. = {q P, de 5r}
(1)
where P is the set of all points, de is the Euclidean distance and 0.6,
The covariance matrix Ep, is:
Ep.
Σ
(p=-p)(p; p)
(2)
1N
ByEN₁;
where p the barycenter of Npp.
The equations for computing the four characteristics are as follows:
Linearity
Lyp serves as an indicator for the presence of linear 1D structure (Weinmann et al.
29
A1,5 A2,i
Lp.
Sphericity S,, provides information regarding the presence of volumetric 3D structure (Wein-
mann et al. 29
A
Sp.
A1,6
Verticality Vp, indicates the property of being perpendicular to the horizon (Hackel et al. 30
LeWos [4] uses it crucial feature because of its high sensitivity to the tree trunk. Here [0 0 1] is the vector the canonical basis.
VP 1- ( ( 1], eg.1)
(5)
PCA1 PCA1p, reflects the slenderness of Np.- The direction of the first eigenvector e basically the most elongated in N,,
A
PCA1,; p: =
(6)
A1,5 + X2, + As,i
Each point endowed now with 12 features encompassing 3 scales, enabling precise depiction of its representation within the local structure. These features are subsequently standardized and integrated into the model's input as an extension to point coordinates for the deep learning model. Refer to Supplementary Section for single-scale and multi-scale ablation studies.

---

Algorithm Geodesic Voxelization Decomposition (GVD)
Input: Voxel grid V; threshold dye; threshold on IER
Output: Voxel partition in / components C = {C
i 11:1
I: i
Component id
2: function GVD(V, T, 7)
3: while V + ( do
4:
C; + 0, ier f- 0, gd + 0
5: Choose 0 e V
6:
while ier < and gd T do
7:
Choose
(())
C(ö) arises from BFS on
8:
gd + d 0, 8)
9:
ier < /ER( 8)
Eq.
10:
C -
end while
12:
C-CUC
i+i41
14:
end while
15:
Return
16:
end function
3.2 Data pre-partitioiing
The GVD algorithm [ is spatial split scheme used to partition the ULS data while preserving the topology of the point cloud. This approach enables the extraction of a of representative training samples from raw forest data, while preserving the local geometry information in its entirety. The point cloud data is first voxelized into a voxel grid V with voxel size equal 0.6 m. an empirical value, and equipped with neighborhood system C. Each voxel e is considered occupied if it contains at least one point. The viteessc-voxelzzation distance dyo between two voxels defined as 1 if they are adjacent, and as the Manhattan distance between and of in voxel space otherwise. This can be expressed mathematically as:
L,
if and of are adjacent in otherwise
(g, v')
x'l 51 2/1.
where (I, y, =) and (z',1/ respectively. We can establish an inter-voxel connectivity information with dge that enables computation of the intrinsic-extrisicc ratio (IER):
v', =²) denote the integers coordinates referring to the index of v and v'
IER(e, v') (y v') do(u, v²)
(8)
We believe that dgo and IER may provide strong cues for the subsequent segmentation task.
Subsequently, we choose one lowest voxel é E V on the vertical z-axis then employ the Breadth-first search (BFS) algorithm to explore its neighbors and the neighbors of neighbors C(û) and calculate the corresponding (g,(ú, ō) and TER( G) where ō E C(6). This procedure terminates when the -oo.esiii-ooelizatiion distance dge exceeds 7 or the IER exceeds 7. T is set to 10 and 7 is set to component C₁ of the
1.5 in this task. All the points within C(t) are extracted and consolidated point clouds (see Figure ((a) and Algorithm The whole procedure is repeated until all the data processed.
As the data partitioned into multiple components, the points within one component C; are further sorted based on their have the same geodesic distance dgn (6, v), where is the lowest voxel inside C; (see Figure (bb) Thus, the GVD algorithm concludes at this stage. Moreover, minimum voxel number and minimum point number within component are configurable, more details are provided in Supplementary Section C
voxelizaaennnnn distance. In this arrangement, points inside one voxel
5

---

3.3 Normalization inside component
Shifting and scaling the point cloud coordinates before training a deep neural network is widely adopted practice that can lead to improved train- ing stability, generalization, and optimization ef- ficiency, see Qi et [26]. Here, prior training the model, the point coordinates are shifted to (0,0,0) and normalized by the longest axis among the three, all the point coordinates being confined within the (0,1) range. This prevents certain co- ordinates from dominating the learning process simply because they have larger values.
go
3.4 Deep neural network training
We employ modified version of PointNet++ MSG for SOUL [8]. The data combination points and features within component is further partitioned into batches of 3,000, which serve as the input for the network. If batch within com- ponent has fewer than 3,000 points, the remaining points randomly chosen from the same compo- nent. The labels of the leaves and wood points are O and respectively. Define B& as such batch, [B 3, 000 and B₁ BR.0 + Ba,1 where By.0 and 1 respectively represent the disjoint sub- batches in B₁ with ground-trus -ruth leaf points and wood points.
(a) Partitioned trunk
(b) Geodesic distance
Figure 3: (a) displays the components found by the GVD algorithm. (b) displays the geodesic distance within the corresponding component. The figures are illustrative, actual UI.S data lacks such depicted complete tree trunks.
Addressing class imbalance issue. Within the labeled ULS training data set, only 4.4% are wood points. The model is overwhelmed by the predominant features of leaves. Therefore, we propose rebalanced loss Ln that changes the ratio of data participating to 1:1 the loss calculation stage by randomly selecting number of leaf points equal to the number of wood points.
In practice, the rebalanced loss is:
LR(Yn,,) - yk log(pk) yk) log(1 pk), ys (Bg, BR,1).
Σ
where YB, specifies the ground truth labels of batch Bx. P E [0, 1] is the model-estimated probability for the label y and By,0 o is defined as
Downsampling(Ba..o (B,,)), if Bk 2 | otherwise.
Be
(10)
BR.
where the downsampling procedure consists of randomly and uniformly selecting subset of size [Ba,1] within Bg.0 without replacement.
The ablation study for the loss function can be found in Supplementary Section D
4 Experiments
We first provide more details about various LiDAR data present each data set. Secondly, the data preprocessing procedure is elaborated upon. Next, delve into the specific configurations of model hyperparameters during the training process, along with the adaptations made to the PointNet++ architecture. Following that, showcase the performance of SOUL the labeled ULS data set. Finally, we demonstrate the generalization ability of SOUL on the other data sets.
4.1 Data Characteristics

---

The ULS data for this study were collected at the Research Station of Paracou French Guiana (85"18' W52"53'). 53'). This tropical forest is situ- ated in the lowlands of the Guiana Shield, the average height of the canopy is 27.2 m with top heights up to 44.8 m (Brede et al. B... Four flights of ULS data were collected at various times over the same site, with each flight cover- ing 14,000 m² of land and consisting of approx- imately 10 million points, yielding a density of around 1, 1,000 p In comparison, the point density of the labeled TLSS data from the same site is about 320,000 pts/m?. Further informa- tion pertaining to LiDAR scanners can be found in Supplementary Section [B]
The labeled ULS data set in French Guiana, where only 4.4% of the points correspond to wood. This significant class imbalance of leaf & wood presents a considerable challenge in discrim- inating wood points from ULS forest point cloud.
Figure 4:
In addition, we used two public data sets with- out labels only for exploring the model's gen- eralization ability. One is a ULS data set from Sandhausen, Germany (Weiser et al. [II, the other one is mobile laser scanning (MLS) data set from New South Wales (NSW), Australia (NSW & Gonzalez (32). The Sandhausen data set uses the same equipment we do, but exhibits lower point density of about 160 pts/m?. The point density of the NSW data set is approximately 53,000 pts/m?.
4.2 Data preprocessing
The first step of the routine involves data cleaning for TLS and ULS. The points exhibiting an intensity lower than -20 dB and those displaying high deviations (Pfennighauer & Ullrich (33) are removed, subsequently the TLS data are subsampled by setting a 0.001 m coordinate precision. The next stage is separate ground by applying cloth simulation filter (CSF) proposed by Zhang et al. (34), which mimics the physical phenomenon of a virtual cloth draped over an inverted point cloud. The ULS data were subjected to similar preprocessing pipeline, except that CSF on ULS (or ALS) performs poorly and was not used. Here we used the software TerraSolid ground detection algorithm on ALS and used the common ground model for all point clouds of the same plot.
4.3 Data annotation
subset of TL.S trees with large trunk labeled using LeWos algorithm (Wang et al. 4 followed by manual refinement process (Martin-Ducup et al. (35)) to improve the outcome accuracy. Then the k-nearest neighbors algorithm (KNN) is employed to assign the tree identifiers (treeIDs) and the labels of TLS points to ULS points. Specifically, for each point in ULS data set, the KNN algorithm selects five nearest points in the TLS data set as reference points and subsequently determines the corresponding label based on the majority consensus of the reference points. It is noteworthy that within TLS data, there exist three distinct label types: unknown, leaf and wood, as demonstrated in Figure I(c]. which respectively correspond to labels: -1, O, and 1. The unknown label is also transferred to ULS points, then such points are filtered out for the next step. Unlabeled and therefore excluded points represent 65% of the ULSS point cloud.
To facilitate model training and quantitative analysis, the labeled ULS data was partitioned into training, validation, and test sets based on treeID, with 221 trees in the training data set, 20 in the validation data set, and 40 in the test data set. This partitioning is visually demonstrated a video where the trees are distinctly grouped together within each data set.
Points per square meter.
The LiDAR device is integrated into backpack, enabling similar scanning process akin TLS.
7

---

4.4 Implementation details
The Adam optimization algorithm is chosen as the optimizer and the learning rate is le-7. Following the practice proposed by Smith et al. [6]. we employ strategy of gradually increasing the batch size by factor of two approximately every 1,000 epochs until reaching final batch size of 128 instead of decreasing the learning rate during the training process. According to our experience, achieving good results requires a substantial duration, typically exceeding 3,000 epochs. In this paper, the prediction results are obtained from checkpoint saved at the 3161st epoch. At this epoch, SOUL achieved Matthews Correlation Coefficient (MCC) value of 0.605 and an AUROC value of 0.888 on the validation data set.
The MCC expression is:
TPXTN FP FN MCC VITP+ + + FP)(TN
(11)
FN)
For binary classification, the calculation of the MCC metric uses all four quantities (TP, TN, FP and FN of the confusion matrix, and produces a high score only if the prediction is good for all quantities (see Yao Shepperd (37), proportionally both to the size of positive elements and the size of negative elements in the data set (Chicco et al. 38
In our task, an output in the form of probability is preferred, it more consistent with the data collection process, since the LiDAR footprint yielding each single echo is likely to cover both wood and leaf elements. However, to facilitate performance comparison with other methods, the model generates discrete outcome that assigns a label to each point. If the probability of a point being classified as leaf exceeds the probability of being classified as wood, it is labeled as leaf and vice versa. Additionally, the probabilistic outputs indicating the likelihood of being a leaf or wood point are retained. More details are provided in the Supplementary Section
4.5 Results on French Guiana ULS data
Comparatively to the prevailing methods em- ployed for forest point clouds, our SOUL ap- proach improves semantic segmentation on ULS forest data by large margins. The results summarized in Table D while Figure 5 high- lights the performance within the tree canopies. In terms of quantitative analysis, SOUL demon- strates specificity metric of 63% in discerning sparse wooden points, where the metric speci- ficity stands for the ratio of true wood points predicted as wood. This value can be elevated to 79%, but at the cost of decreasing the accuracy metric to 80%, where the metric accuracy stands for the ratio of correctly predicted wood and leaf points. For better understanding how each class contributes to the overall performance, we introduced two more informative metrics mentioned in the study of Branco et al. [39], Geometric mean (G-mean) and Balanced Accuracy (BA). These metrics validate the performance of the SOUL model, providing more comprehensive insights into its capabilities.
(a) FSCT
(b) SOUL
(c) Ground truth
Figure 5: Qualitative results on ULS test data.
Because of the class imbalance issue, methods such as FSCT LeWos and other existing ap- proaches developed for dense forest point clouds, like TLS or MLS, (cf. Section 2 are ineffective.
4.6 Results on the other data.
After testing on data sets from Sandhausen data set (Weiser al. 31 and NSW Forest data set 32 we observed that the SOUL exhibits good generalization capability across various forests. Qualitative results are shown in Figure 6 First, SOUL. demonstrates good results on Sandhausen data set [3]. However, SOUL does not outperform the others methods on TLS, but it shows better discrimination within the tree canopy. A possible improvement of SOUL performance is foreseen for denser data provided the model is retrained on similar data sets.
'TP, TN, FP, EN stand respectively True Positives, True Negatives, False Positives, False Negatives.

---

Discussion
GVD as a spatial split scheme. Semantic segmentation in tropical forest environments presents distinctive challenges that differ from objects with regular shapes found in data sets such as ShapeNet (Chang et al. [40]) and ModelNet40 (Wu et al. (41). Forest point clouds are inherently poorly structured and irregular because leaves and branches are small with regard the LiDAR footprint size, a lot of points derived from LiDAR device are mixed between the two materials of leaves and wood. We propose GVD, ad-hoc method for UL.S data, to overcome the chaotic nature of forest point clouds with pre-partition step. Similar to the enhancement that RCNN (Girshick et al. 442) brings CNN (LeCun al. [43 on image classification and semantic segmentation tasks, GVD serves as a region proposal method that provides improved training sample partitioning for deep learning networks. The neural network can benefit from more refined and informative data, leading to enhanced performance in the semantic segmentation task.
0.75
0.3
0000
(a) Paracou, French Guiana TLS
(b) NSW, Australia MLS
(c) Sandhausen. Germany ULS
Figure 6: Qualitative results on various LiDAR data from different sites.
Rebalanced loss designed for class imbalance issue. In our model, PointNet++ is chosen as the backbone network due to its known robustness and effectiveness in extracting features from point clouds (Qian al. [44] ). But learning from imbalanced data in point cloud segmentation poses persistent challenge, as discussed by Guo et al. [45]. To address this issue, we employ a rebalanced loss function tailored this task. Rather than adjusting class weights during training (Lin al. [[]; Sudre et al. [46]), we adopt random selection approach to balance the training data during the loss computation. This decision stems from the observation that misprediction of subset of the minority class can lead to significant fluctuations in the loss value, consequently inducing drastic changes in gradients. an extreme scenario, wherein single point represents a positive sample, correct prediction of this point drives the loss towards zero, irrespective of other point predictions. Conversely, misprediction of the single point yields a loss value approaching one.
Table 1: Comparison of different methods
Methods
Accuracy
Recall
Precision
Specificity
G-mean
BA
FSCT
0.974
0.977
0.997
0.13
0.356
0.554
FSCT retrain
0.977
1.0
0.977
0.01
0.1
0.505
LeWos 4
0.947
0.97
0.975
0.069
0.259
0.520
LeWos (SoD2)
22
0.953
0.977
0.975
0.069
0.260
0.523
SOUL (focal loss 13
0.942
0.958
0.982
0.395
0.615
0.677
SOUL (rebalanced loss)
0.826
0.884
0.99
0.631
0.744
0.757
BA (Balanced Accuracy) BA = (Recall + Specificityy)
SoD (Significance Difference).

---

SOUL model's generalization capability. Despite being originally developed for tropical forest environments, SOUL demonstrates promising performance when applied to less complex forests. We anticipate that with the increasing prevalence of ULS data, coupled with more high-quality labeled data sets, the performance of SOUL. poised to advance further. Additionally, there is potential for SOUL. to be extended to other complicated LiDAR scenes. It is conceivable to envision the development of a universal framework based on SOUL that can effectively handle various types of forest LiDAR data, including ULS, TLS, MLS, and even ALS.
Ablation study of geometric features. An ablation study is conducted to evaluate the impact of individual geometric features proposed in our model. Comparative experiments, as detailed in Figure 7 Table illustrate the advantages of utilizing multiple precomputed geometric features at various scales. Specifically, we observe that features like linearity and verticality significantly enhance trunk recognition, while PCA1 and Sphericity are effective within the canopy. While PointNet++ possesses the capability internally learn complex features, it may not precisely replicate empirically selected geometric attributes. PointNet++ excels at adapting and acquiring sophisticated features from training data, whereas our predefined geometric features offer specific advantages model's context.
Table 2: Comparison of single geometric feature and multiple geometric features
Methods
Accuracy
Recall
Precision
Specificity
G-mean
BA
SOUL (linearity + one scale)
0.962
0.978
0.983 0.99
0.31 0.631
0.55
0.636
SOUL (unabridged)
0.826
0.884
0.744
0.757
(a) Ground Truth
(b) Only linearity
(c) Only linearity
(d) Full version
(e) Full version
Figure 7: Comparing the unabridged model to a downgraded model using a single-scale feature linearity. Figures (c) and (e) represent different perspectives of Figures (a), (b), and (d).
The current observation is that SOUL may not perform as well as other methods on TLS data, was not trained for that. Additionally. SOUL may not perform effectively for trees that significantly deviate from the training data. By incorporating diverse samples during training, the model can mitigate the challenges associated with vegetation disparity.
Limitations.
Conclusion
We present SOUL, novel approach for semantic segmentation in complex forest environments. It outperforms existing methods in the semantic segmentation of ULS tropical forest point clouds and demonstrates high performance metrics on labeled ULS data and generalization capability across various forest data sets. The proposed GVD method is introduced as spatial split schema to provide refined training samples through pre-partitinn martiiiiion. One key aspect of SOUL is the of the rebalanced loss function, which prevents drastic changes gradients and improves segmentation accuracy. While SOUL shows good performance for different forest types, it may struggle with significantly different trees without retraining. Future work can focus on improving the performance of SOUL on denser forest point clouds broaden its applications.
Acknowledgement and Disclosure of Funding
The authors are grateful to Nicolas Barbier and Olivier Martin-Ducup for their contributions in data collection and TLSS data preprocessing. We also acknowledge that this work has been partially supported by MIAI @ Grenoble Alpes, (ANR-19-P31A-0003)
10

---

References
[1] G. Vincent, C. Antin, M. Laurans, J. Heurtebize, S. Durrieu, C. Lavalley, and J. Dauzat, "Mapping plant index of tropical evergreen forest by airborne laser scanning. A cross- validation study using LA12200 optical sensor," Remote Sensing of Environment, vol. 198, pp. 254-266, 2017.
[2] A. Ullrich and M. Pfennigbauer, "Categorisation of full waveform data provided by laser scanning devices," in Electro- Remote Sensing, Photonic Technologies, and Applications (G. J. Bishop, D. Gonglewski, and G. W. Kamerman et al., eds.), vol. 8186, p. 818609, International Society for Optics and Photonics, SPIE, 2011.
[3] B. Brede, H. M. Bartholomeus, N. Barbier, F. Pimont, G. Vincent, and M. Herold, "Peering through the thicket: Effects of UAV LiDAR scanner settings and flight planning on canopy volume discovery," International Journal of Applied Earth Observation and Geoinformation. vol. 114, 103056, 2022.
[4] D. Wang, S. Momo Takoudjou, and E. Casella, "LeWoS: A universal leaf-wood classification method to facilitate the 3D modelling of large tropical trees using terrestrial LiDAR," Methods in Ecology and Evolution, vol. 11, no. 3, pp. 376-389, 2020.
[5] G. Vincent, Verley, B. Brede, G. Delaitre, E. Maurent, Ball, Clocher, and N. Barbier, "Multi-sensor airborne lidar requires intercalibration for consistent estimation of light attenuation and plant area density," Remote Sensing of Environment, vol. 286, p. 113442, 2023.
[6] B. Wu, G. Zheng, and Y. Chen, "An Improved Convolution Neural Network-Based Model for Classifying Foliage and Woody Components from Terrestrial Laser Scanning Data." Remote Sensing, vol. 12, no. 6, 2020.
[7] J. Morel, A. Bac, and T. Kanai, "Segmentation of Unbalanced and In In-Homogeneous Point Clouds and Its Application to 3D Scanned Trees," " Vis. Comput., vol. 36, p. 2419-2431, oct 2020.
[8] C. Qi, L. Yi, Su, and L. J. Guibas, "PointNet++: Deep Hierarchical Feature Learning on Point Sets in Metric Space," in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, Curran Associates, Inc., 2017.
[9] S. Krisanski, M. S. Taskhiri, S. Gonzalez Aracil, D. Herries, A. Muneri, M. B. Gurung, J. Montgomery, and P. Turner, "Forest Structural Complexity Tool-An Open Source, Fully- Automated Tool for Measuring Forest Point Clouds," Remote Sensing, vol. 13, no. 22, 2021.
[10] L. Windrim and M. Bryson, "Detection, Segmentation, and Model Fitting of Individual Tree Stems from Airborne Laser Scanning of Forests Using Deep Learning," " Remote Sensing, vol. 12, no. 9, 2020.
[11] T. He, H. Huang. Yi, Y. Zhou, C. Wu, Wang, and S. Soatto, "GeoNet: Deep Geodesic Networks for Point Cloud Analysis." " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6881-6890, June 2019.
[12] M. Liu, X. Zhang, and H. Su, "Meshing Point Clouds with Predicted Intrinsic- Ratio Guidance," in European Conference on Computer Vision (ECCV), Glasgow, UK (A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, eds.), (Cham), pp. 68-84, Springer International Publish- ing, August 2020.
[13] T.-Y. Lin, Goyal, R. Girshick, He, and Dollár, "Focal Loss for Dense Object Detection," " IEEE Transactions on Puttern Analysis and Machine Intelligence, vol. 42, no. 2, pp. 318-327, 2020.
[14] Y. Bai, G. Vincent, N. Barbier, and O. Martin-Dueup, "UVA laser scanning labelled las data over tropical moist forest classified leaf or wood points," Oct 2023.
[15] D. Wang, "Unsupervised semantic and instance segmentation forest point clouds," ISPRS Journal of Photogrammetry and Remote Sensing, vol. 165, pp. 86-97, 2020.
11

---

[16] L. Landrieu and M. Simonovsky, "Large-Scale Scale Point Cloud Semantic Segmentation with Super- point Graphs." in 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4558 4567, 2018.
[17] S. M. Krishna Moorthy, K. Calders, M. B. Vicari, and H. Verbeeck, "Improved Supervised LearningBBased Approach for Leaf and Wood Classification From LiDAR Point Clouds of Forests," IEEE Transactions on Geoscience and Remote Sensing, vol. 58, no. 5, pp. 3057-3070, 2020.
[18) G. Zheng, L. Ma, W. He, J. U. H. Eitel, L. M. Moskal, and Z. Zhang, "Assessing the Contribution of Woody Materials to Forest Angular Gap Fraction and Effective Leaf Area Index Using Terrestrial Laser Scanning Data," IEEE Trunsactions Geoscience and Remote Sensing, vol. 54, no. 3, pp. 1475-1487, 2016.
[19] J.-F. Lalonde, N. Vandapel, and M. Hebert, "Automatic Siree-Dimensionl Point Cloud Pro- cessing for Forest Inventory," Tech. Rep. CMU-RI-TR-00-21,, Carnegie Mellon University, Pittsburgh, PA, July 2006.
[20] K. Itakura, S. Miyatani, and F. Hosoi, "Estimating Tree Structural Parameters via Automatic Tree Segmentation From LiDAR Point Cloud Data," IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 15, pp. 555 555-564, 2022.
[21] M. B. Vicari, M. Disney, P. Wilkes, A. Burt, K. Calders, and W. Woodgate, "Leaf and wood classification framework for terrestrial LiDAR point clouds," Methods in Ecology and Evolution, vol. 10, no. 5, pp. 1800699, 2019.
[22] P. Wan, Shao, Jin, T. Wang. S. Yang, G. Yan, and W. Zhang, "A novel and efficient method for wood-leaf separation from terrestrial laser scanning point clouds at the forest plot level," Methods in Ecology and Evolution, vol. 12, no. 12, pp. 2473-2486, 2021.
[23] X. Shen, Q. Huang, X. Wang, J. Li, and B. Xi, "A Deep Learning Based Method for Extracting Standing Wood Feature Parameters from Terrestrial Laser Scanning Point Clouds of Artificially Planted Forest," Remote Sensing, vol. 14, no. 15, 2022.
[24] Li R. Bu, M. Sun, W. Wu, X. Di, and B. Chen, "PoinCCNN: Convolution On X- &rannsformed Points," in Advances in Neural Information Processing Systems (NeurIPS), vol. 31, Curran Associates, Inc., 2018
[25] S. Ren, K. He, R. Girshick, and J. Sun, "Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks." in Advances in Neural Information Processing Systems (NeurIPS), NIPS' 15, (Cambridge, MA, USA), p. 91-99, MIT Press, 2015.
[26] C. Qi, H. Su, K. Mo, and L. J. Guibas, "PointNet: Deep Learning on Point Sets for 3D the
Classification and Segmentation." in Proceedings the IEEE Conference Computer Vision and Pattern Recognition (CVPR), July 2017.
[27] J. Choe, C. Park, F. Rameau, J. Park, and I. S. Kweon, "PointMixer: MLP-Mixer For Point Cloud Understanding." " in European Conference on Computer Vision (ECCV), 2022.
[28) H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V. Koltun, "Point transformer," in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 16259-16268, 2021.
[29] M. Weinmann, B. Jutzi, and C. Mallet, "Feature relevance assessment for the semantic inter- pretation of 3D point cloud data," ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, vol. II-5/W2, pp. 313-318, 2013.
[30] T. Hackel, J. D. Wegner, and K. Schindler, "Contour Detection in Unstructured 3D Point Clouds," in Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), pp. 1610-1618, 2016.
[31] H. Weiser, L. Winiwarter, V. Zahs, P. Weiser, K. Anders, and B. Höfle, "UAV- PhAy-Phhtogrammmetry UAV laser scanning and terrestrial laser scanning point clouds of the inland dune in Sandhausen, Baden- Baden-Wirttembeed Württemberg, Germany," 2022.
12

---

[32] Natural Resources Commission NSW Government and S. Gonzalez, "NSW forest mon- itoring and improvement program forest plot network ground based Lidar pilot," 2022. https://portal.ter..og htt.:/a.......oraaa//aaaaaaa//a/ a
[33] M. Pfennigbauer and A. Ullrich, "Improving quality of laser scanning data acquisition through calibrated amplitude and pulse deviation measurement," in Laser Radar Technology and Appli- cations XV (M. D. Turner and G. W. Kamerman, eds.), vol. 7684 of Society of Photo-Optical Instrumentation Engineers (SPIE) Conference Series, p. 76841F, Apr. 2010.
[34] W. Zhang. J. Qi, P. Wan, H. Wang, D. Xie, X. Wang, and G. Yan, "An Easy-to- tose Airborne LiDAR Data Filtering Method Based on Cloth Simulation," " Remote Sensing, vol. 8, no. 6, 2016.
[35] O. Martin-Ducup, I. Mofack, Gislain, D. Wang, P. Raumonen, P. Ploton, B. Sonké, N. Barbier, Couteron, and Pélissier, "Evaluation of automated pipelines for tree and plot metric estimation from TLS data in tropical forest areas," Anmals of Botany, vol. 128, pp. 753-766, 04 2021.
[36] S. L. Smith, P.-. Kindermans, and Q V. Le, "Don't Decay the Learning Rate, Increase the Batch Size," in International Conference on Learning Representations (ICLR), 2018.
[37] Yao and M. Shepperd," "Assessing software defection prediction performance." in Proceedings of the Evaluation and Assessment in Software Engineering, ACM, apr 2020.
[38] D. Chicco and G. Jurman, "The advantages of the Matthews correlation coefficient (MCC) over
F1 score and accuracy in binary classification evaluation," BMC Genomics, vol. 21, 01 2020.
[39] P. Branco, L. Torgo, and R. P. Ribeiro, "A Survey of Predictive Modeling Imbalanced Domains," ACM Compur. Surv, vol. 49, aug 2016.
[40] A. X. Chang. T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi, and F. Yu, "ShapeNet: An Information-Rian 3D Model Reposi- tory." 2015.
[41] Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, "3D ShapeNets: A Deep Representation for Volumetric Shapes," 2015.
[42] R. Girshick, J. Donahue, T. Darrell, and J. Malik, "Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation," " in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 580-587, 2014.
[43] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, "Backpropagation Applied to Handwritten Zip Code Recognition," Neural Comput., vol. 1, p. 55.4-151, dec 1989.
[44] G. Qian, Y. Li, H. Peng, J. Mai, H. Hammoud, M. Elhoseiny, and B. Ghanem, "Po Revisiting PointNet++ with Improved Training and Scaling Strategies," in Advances in Neural Information Processing Systems (NeurIPS), 2022.
[45] Y. Guo, H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun, "Deep Learning for 3D Point Clouds: A Survey," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PP, pp. 1-1, 06 2020.
[46] C. H. Sudre, W. Li, T. Vercauteren, S. Ourselin, and M. J. Cardoso, "Generalised Dice Overlap as Deep Learning Loss Function for Highly Unbalanced Segmentations." in Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support, pp. 240-248, Springer International Publishing, 2017.
[47] J. Wojtanowski, M. Zygmunt, M. Kaszczuk, 7. Mierezyk, and M. Muzal, "Comparison of 905 nm and 1550 nm semiconductor laser rangefinders' performance deterioration due to adverse environmental conditions," Opto-Electronics Review, vol. 22, no. 3, pp. 183-190, 2014.
[48] D. Kingma and J. Ba, "Adam: Method for Stochastic Optimization," " in International Conference Learning Representations (ICLR), 2015.
13

---

[49] C. Liu and M. Belkin, "Accelerating SGD with momentum for paermmarameeerieed learning," in International Conference on Learning Representations (ICLR), 2020.
[50] D.-A. Clevert, T. Unterthiner, and S, Hochreiter, "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs).." in International Conference on Learning Representations, 2016.
[51] V. Nair and G. E. Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," in International Conference Machine Learning. 2010.
[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. Vitioo, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang. Bai, and S. Chintala, "PyTorch: An Imperative Style, High-Perie Perforreffoe Deep Learning Library," in Advances in Neural Information Processing Systems (NeurIPS) 32, pp. 8024 8035, Curran Associates, Inc., 2019.
14

---

Supplementary
A Overview
In this supplementary material, we provide more quantitative results, technical details and additional qualitative test examples. Section B presents more implementation and training details of SOUL, in Section the efficacy of the GVD algorithm is discussed, while Section D conducts ablation studies to investigate the impact of different loss functions on the model's performance. At last, Section E demonstrates the performance improvement achieved by employing multiple-scale computation of geometric features compared to single-scale approach. A video showcasing the performance of SOUL is available at this link: here
B Additional Implementation Details
As supplement to Sections 3.4 and 4.4, we provide additional details here.
LiDAR scanner. Table presents summary of the distinguishing laser characteristics between ULS and TLS. The acquisition LiDAR data is profoundly impacted by atmospheric characteristics, the LiDAR extinction coefficients exhibit a low impact to atmospheric humidity at both 905 nm and 1550 nm wavelengths (Wojtanowski et al. (47). ULS uses 905 nm as wavelength, at which the scanning device hast the minimum energy consumption (Brede et al. [3). Meanwhile, the TLS system operates at a wavelength of 1550 nm, which is more susceptible to the fog impact (Wojtanowski al. (47)) This poses the issue that the spectral reflectance of leaf and wood is less contrasted (Brede et al. ( at the 905 nm wavelength, thereby leading us to employ only point coordinates input for avoiding the use of intensity information.
Table 3: Laser sensor characteristics
TLS
ULS
RIEGL Scanner
VZ400
MiniVux-IUUA
Laser Wavelength (nm)
1550
905
Beam divergence (mrad)
0.25
1.6*0.5
Footprint diameter (cm@) 100m)
3.5
1645
Pulse duration (ns)
3
6
Range resolution (m)
0.45
0.9
Training details. In addition to the content already illustrated in the main paper, we provide further details regarding the training parameters. First and foremost, sufficient training time required for the model to achieve desirable performance. Our observation indicates that models trained less than 2,000 epochs are inadequately trained to achieve desirable performance. During the training process, three metrics, AUROC, MCC and Specificity, Especially, the metric of specificity is crucial as it measures the ability to accurately discriminate wooden points, which is the primary requirement of our method. Another thing note is that, contrary to the mentioned practice in the main text increasing the batch size by a factor of two every 1,000 epochs, we did not strictly follow this restriction during the actual training process. Often, the increase in batch size occurred around 800-900 or 11001200 1200 epochs for better using the GPU resources. However, theoretically, this offset should not affect the final performance.
considered more informative and insightful.
Architecture of DL model. We have made several modifications to the architecture of PointNet++ (MSG) (8). Firstly, we observed that Adam (Kingma & Ba (48)) outperformed Nesterov SGD (Liu & Belkin [49]) in this task, and ELU (Clevert et al. (50) activation function was better than ReLU (Nair & Hinton (5T). The fraction of the input units to drop for dropout layer was changed from 0.5 to 0.3, that means the probability of an element to be zeroed is 30% (Paszke et al. (52). We also decreased the number of hidden neurons and added two more fully connected layers at the end.
Test with error bar. We have calculated a confidence interval at 95% confidence level to indicate the uncertainty of our method. In the main body of the paper, it was mentioned that we obtained
15

---

data from four flights, and each data collection from these flights allowed us to obtain a labeled ULS data set through label transfer from TLS data. The test data set composed by 40 trees from the same positions across these four labeled ULS data sets, resulting in total of 160 trees. We divided further these 160 trees based on their unique treell) to 16 smaller test sets. However, trees with the same treell are not placed together in the same test set. The calculation of confidence intervals was derived from these test sets, The result is summarized in Table and Table[ It is clear that SOUL outperforms all other methods on ULS data and achieves state-of-the-art performance on metrics such as Specificity, Balanced Accuracy and G-mean.
Table 4: Comparison of different methods
Methods
Specificity
G-mean
BA
FSCT [9]
0.13
0.356
0.554
FSCT retrain
0.01
0.1
0,505
LeWos 4
0.069
0.259
0.520
LeWos (SoD*)
22 [3]
0.069
0.260
0.523
SOUL (focal loss
0.395
0,615
0.677
SOUL (rebalanced loss)
0.576 + 0.063
0.651 + 0.030
0.720 + 0.027
BA (Balanced Accuracy) BA (Recall + ficity.......
SoD (Significance of Difference).
Table Comparison of different methods
Methods
Accuracy
Recall
Precision
FSCT
0,974
0.977
0.997
FSCT retrain
0.977
1.0
0.977
LeWos
4
0,947
0.97
0.975
LeWos (SoD)
22
0.953
0.977
0.975
SOUL (focal loss [6]
0.942
0.958
0.982
SOUL (rebalanced loss)
0.857 + 0.014
0.865 = 0.015
0.988 + 0.002
Output contains redundant points. To proceed with further data analysis and usage, it is necessary remove duplicate points present in the output.
C
Assessing the efficacy of GVD
The ULS data often covers several hectares, or even hundreds of hectares. The situation in tropical forests is also highly complex, with various types and sizes of trees densely packed together. The significant memory demand makes it nearly impossible to process all the data at once, leading us to adopt spatial split scheme approach
necessary compromise.
We can select data randomly from the whole scene, but selecting data randomly can result in a sparse and information-pon sample. An alternative is to employ divide and conquer strategy to handle the chaotic, big volume, and complex ULS data. That is why we propose GVD, a method that involves breaking down the data into more manageable subsets (see Figure [8. allowing to handle the intricacies and extract meaningful insights in a more systematic manner. This approach enables us to retain the information-rie aspects of the data while overcoming computational challenges associated with the sheer volume of data.
Prior to employing the GVD method, we initially adopted a more intuitive approach. The data was partitioned in unit of voxel, serving as component for batch preparation through nown-sampling However, this approach gave rise to border effects, particularly impeding SOUL's focus on the meticulous segmentation of intricate branch and leaf within tree canopy (see Figure [c)) The segmentation of cubes led to the emergence of noise point clusters along the voxel edges. ULS have more points on tree canopy, so the presence noise point clusters on voxel edges is more severe on tree canopy, which imposes bigger obstacles to our leaf/wood segmentation task.
16

---

(a) Comp 1148
(b) Comp Nº575
(c) Comp N91137
(d) Comp No552
(c) Comp N*1148
(f) Comp 5
(g) Comp N N111
(h) Comp N
Figure 8: The figure depict distinct samples subjected to GVD processing and presents qualitative results within each sample. The upper row illustrates the ground truth for each sample (where brown represents leaves and white represents wood), whereas the lower row exhibits the predictive results generated by SOUL (where blue represents leaves and red represents wood incorporating gradient colors for transitional probability).
(a) Top view
Middle layer
(C) Thin layer
(d) Back view
Figure 9: Downgrade version's output with sliding window. The use of GVD eliminates observed edge effects in this context.
We experimented with cuboids, choice aimed preserving greater semantic information within each component sample and expanding the spatial range for batch selection. Similar "sliding window", we can systematically traverse the entire forest with overlapping coverage in this way. But border effects persisted (see Figure 9.. prompting the introduction of the GVD method, which led to substantial improvement. Modifying parameters and y adjusts the coverage scope of each component in GVD segmentation. Additionally, tweaking the minimum accepted number of voxels and points, two GVD's configurable thresholds, within each component effectively manages component size. This process aids in the elimination of low-information content outlier point clusters to certain extent.
comparison between the result of the downgraded version using sliding window as spatial split schema and the individual-compones point performance of full version SOUL is illustrated in Figure Notably, in component 552, SOUL. effectively discriminates trunk points from leaf points that previously entirely intertwined, surpassing initial expectations.
17

---

Cross entropy loss
(b) Focal loss
Figure 10: The predictions of SOUL. model on raw ULS data training with cross entropy loss function and focal loss function. Blue represents wood points with high probability, while red represents Ieaf points with high probability. Despite utilizing the focal loss, the model remains overwhelmed by leaf' points.
D Ablation study of rebalanced loss
In this section, we mainly discuss the significant benefits introduced by the rebalanced loss. The network is biased towards Ieaf points when using cross-entropy loss function and the use of focal loss function (Lin et al. (T3 does not yield substantial improvements to the task, as its performance remains unsatisfactory when tested on the raw data set (see Figure T0(a) and Figure 10(b)
Through a comparison of the specificity curves (see focal loss specificity in Figure II versus rebalanced loss specificity in Figure [2) and the training/validatiin rebalanced loss (see loss curve of the focal loss in Figure 13 versus loss curve of the rebalanced loss in Figure 14). we observed that focal loss failed to address the issue of class imbalance in task. Plus, it important to note that in the loss curve of rebalanced loss, the validation loss curve showed significant fluctuations Figure 14 As foreseen, this variation occurred because the training process employed the rebalanced loss for model training, while the validation process utilized the cross-entropy loss function. Consequently, the loss value used for backpropagation is derived from the rebalanced loss function, which inherently does not consider the majority of leaf points. So once using cross entropy calculate the loss, all points within batch are taken into account, which can lead to fluctuations in the validation loss curve. However, it was expected that the validation loss curve would eventually converge on the validation set to demonstrate that using rebalanced loss backward propagation effectively balances the representation of leaf and wood points. In fact, the final results surpassed focal loss by a significant margin, as evidenced by the convergence of validation loss curve for rebalanced loss (see Figure For the loss curve of focal loss in Figure T3 both the training and validation processes employ the focal loss function.
loss curves of focal loss and
0.5
ww
Figure 11: Specificity focal loss (single-scale feature calculation). In the figure, the cyan curve represents the specificity curve of the training data set, while the gray curve represents the specificity curve of the validation data set.
18

---

03
03
Figure 12: Specificity rebalanced loss (single-scale feature calculation), cyan curve training data set, gray curve validation set. set.
363
263
34
Figure 13: Loss focal loss (single-scale feature calculation). The blue curve represents the loss curve of the training data set, while the red curve represents the loss curve of the validation data set.
Figure 14: Loss rebalanced loss (single-scale feature calculation), blue curve trrininng data set, red curve validation data set.
Figure 15| Figure 16 and Figure 17 we can clearly observe the presence of 'sarrp fluctuations" which are result of the operation mentioned on main paper and Section where increased the batch size by a factor of approximately every 1,000 epochs. We followed this practice proposed by Smith et al. [36], which involves increasing the batch size instead of decaying the learning rate.
In Figure 12 Figure 14
E Single-scale vs Multiple-scale
In this section, mainly showcase multiple-scale geometric features calculation outperform single- scale at our task. After applying the rebalanced loss as the loss function, we observed that computing geometric features multiple scales ultimately improved the performance of SOUL. When comparing the evolution specificity with the number of epochs between single-scale and multiple-scale (see Figure [6] and Figure I7p. multiple-scale not only exhibit higher values but also converge more effectively over time, same for loss convergence (see Figure [4 Figure 15).
The Matthews Correlation Coefficient (MCC) (see Yao & Shepperd 377 and AUROC are both effective metrics to evaluate the model performance under class imbalance. Therefore, besides the specificity, we provide the MCC and AUROC values for both single-scale and multiple-scale cases
19

---

Figure 15: Loss rebalanced loss (multiple-scale feature calculation), blue curve training data set, red curve validation data set.
02
Figure 16: Specificity Single-scale geometric features calculation, cyan curve training data set, gray curve validation data set.
08
03
Figure 17: Specificity Multiple-scale geometric features calculation, cyan curve training data set, gray curve validation data set.
during the training process. Upon analyzing the comparative results of MCC in Figure 18 and Figure [9] and AUROC in Figure and Figure we consistently observe that the multiple-scale computation of geometric features output higher values compared to the single-scale. This strongly supports the superior performance of multiple-scale feature computation over single-scale.
040
Figure 18: During the training process, the MCC (single-scale) values change with the number of epochs.
20

---

3.
Figure 19: MCC (multiple-scale) values change with the number of epochs.
055
Figure 20: AUROC (single-scale) values change with the number of epochs.
055
Figure 21: AUROC (single-scale) values change with the number of epochs.
MCC and AUROC provide valuable insights for selecting the final model state (checkpoint). For example, up now, the berf-oerg model state is the obtained the 3161st epoch with the multiple-scall geometric features computation, as mentioned in the main paper. The model achieves MCC value of 0.605 and an AUROC value of 0.888, these results generally outperform all single-scale metrics. As MCC is a discrete case of Pearson correlation coefficient, we can conclude that there exists a strong positive relationship exists between the model's predictions and the ground truth by using the interpretation of Pearson correlation coefficient.
21


Reviews:
[{"review_id": "HXfIH4LLHF", "reviewer": "Reviewer_eT4d", "summary": "This submission proposes an end-to-end approach, Semantic segmentation On ULS (SOUL), for leaf-wood semantic segmentation that is based on PointNet++ [8]. By considering the imbalanced class label in the collected ULS dataset, a rebalanced loss is used. Moreover, a geodesic voxelization decomposition (GVD) method is introduced for data refinement through pre-partition. A ULS dataset with 282 tree-labels is collected for network training and testing. Experiments on the collected dataset demonstrate the effectiveness of the proposed method compared with the chosen baselines.", "questions": "See Weaknesses.", "limitations": "See Weaknesses.", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "This submission proposes an end-to-end approach, Semantic segmentation On ULS (SOUL), for leaf-wood semantic segmentation that is based on PointNet++ [8]. By considering the imbalanced class label in the collected ULS dataset, a rebalanced loss is used. Moreover, a geodesic voxelization decomposition (GVD) method is introduced for data refinement through pre-partition. A ULS dataset with 282 tree-labels is collected for network training and testing. Experiments on the collected dataset demonstrate the effectiveness of the proposed method compared with the chosen baselines.\n\nThe strengths are:\n1) A new dataset has been collected.\n2) A new approach with comparable results.\n", "weaknesses": "The weaknesses of this paper are listed as follows:\n\n1) The writing and the organization of the submission need to be improved. \n\n2) The benefits of using the data pre-partitioning (in Section 3.2) are not clear. It would be better to provide more details and an ablation study w/o the GVD method.\n\n3) The details of the provided baselines are missing. It would be better to consider more baselines in Table 1, e.g., PointNet++ with the proposed sub-modules.\n\n4) The novelty is not sufficient for NeurIPS standards.\n\n5) There are lots of approaches for imbalanced data labels. It would be better to provide more experiments to demonstrate the effectiveness of the rebalanced loss used.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable and constructive comments, we appreciate this opportunity to respond to your comments and address your concerns.\n\nQ1: The writing and the organization of the submission need to be improved.\n\nWe will improve both in the revised version and check carefully and correct all typos.\n\nQ2: The benefits of using the data pre-partitioning (in Section 3.2) are not clear. It would be better to provide more details and an ablation study w/o the GVD method.\n\nFurther elaboration on the benefits of the GVD method is warranted in our paper. ULS data often covers several hectares, or even hundreds of hectares. The situation in tropical forests is also highly complex, with various types and sizes of trees densely packed together. The significant memory demands make it nearly impossible to process all the data at once, leading us to adopt a spatial split scheme approach as a necessary compromise. \n\nWe can select data randomly from the whole scene, but selecting data randomly can result in a sparse and information-poor sample. An alternative is to employ a divide and conquer strategy to handle the chaotic, big volume, and complex ULS data. That's why we propose GVD, a method that involves breaking down the data into more manageable subsets (refer to Figure 1 in rebuttal_figures.pdf), allowing us to handle the intricacies and extract meaningful insights in a more systematic manner. This approach enables us to retain the information-rich aspects of the data while overcoming computational challenges associated with the sheer volume of data.\n\nIn our global response to all the reviewers, we have introduced an alternative spatial split scheme called the \"sliding window\". This method involves dividing the entire scene into multiple overlapping cuboids. Within each cuboid, we select training, validation, and testing samples. While this isn't a strict ablation study of GVD, this allows us to observe the clear impact of border effects and the disruption of spatial information in point cloud data (refer to Figure 2 in rebuttal_figures.pdf). This demonstrates the advantages of the GVD method.\n\nQ3: The details of the provided baselines are missing. It would be better to consider more baselines in Table 1, e.g., PointNet++ with the proposed sub-modules.\n\nIn fact, SOUL is the first approach to tackle semantic segmentation of tropical forest ULS data. SOUL might serve as the first baseline in this field. We have introduced both the best-performing unsupervised method (LeWoS) and a deep learning approach (FSCT) tailored for TLS data. The latter are commonly recognized as the best alternatives in the context we are dealing with (semantic segmentation of forest point clouds). These methods do not perform well on ULS data, mainly because they were not specifically designed to handle the raw ULS data and address class imbalance issues. Our qualitative evaluation has demonstrated that our methods represent the state-of-the-art in terms of performance. \n\nIt is worth noting that access to code is limited in this domain, hindering reproducibility and comparative analysis. Moreover, certain methods rely on intensity data, which can vary based on different devices. Consequently, the generalizability of testing results remains uncertain. That's the reason why we aim at using solely coordinates as input of SOUL. To foster collaboration and facilitate meaningful exchange within the research community, we have made the decision to openly share our code and data, hoping to engage with fellow researchers in a more inclusive and productive manner.\n\nQ4: The novelty is not sufficient for NeurIPS standards.\n\nWe appreciate the reviewer's feedback and understand their concern regarding the novelty of our work. Nonetheless, we would like to highlight that our research does indeed bring valuable contributions to the field. The proposed method GVD is also relevant for other LiDAR sensors and type of forests and the rebalanced loss addresses the challenge of class imbalance in real-world data, which may have practical implications in various domains.\n\nAs mentioned in the global response and in the response to reviewers yADz and bCLj, the innovative parts of our proposal lie in (1) a whole pipeline for the forest segmentation task with potential community impact, (2) GVD preprocessing for sparse forest data and (3) rebalanced loss function designed for unbalanced ULS forest recordings. We will try to better emphasize these aspects in our revisions.\n\nQ5: There are lots of approaches for imbalanced data labels. It would be better to provide more experiments to demonstrate the effectiveness of the rebalanced loss used.\n\nWe have provided a more comprehensive comparison between the focal loss and the rebalanced loss in our global response to all reviewers and in the final version. Additionally, we will publish our code and data, hoping for further interactions with researchers in the field. "}, {"review_id": "Hcbb4fgUVf", "reviewer": "Reviewer_uJjq", "summary": "This paper introduces a neural network model based on the Pointnet ++ architecture which makes use of point geometry only (excluding any spectral information). To cope with local data sparsity, it proposes a sampling scheme that aims to preserve local important geometric information. It also proposes a loss function adapted to the severe class imbalance. Experiments show that the proposed model outperforms state-of-the-art methods on UAV point clouds.", "questions": "Probably need to go beyond the pointnet++ methods which is relatively outdated.", "limitations": "The innovation part seems lacking.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper applies the mature pointnet++ on lidar tree classification with some variations of the methodology. The results seems good and working on the low resolution UAV LIDAR point clouds.", "weaknesses": "I am not quite sure of the novelty part as most of the techniques seem mature technique.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comments and  advice. We appreciate the opportunity to address your concerns.\n\nQ1: Probably need to go beyond the pointnet++ methods, which is relatively outdated.\n\nIn fact, the recent work by Qian et al.[1] at NeurIPS 2022 demonstrates PointNet++ backbone’s enduring relevance. They achieved substantial performance gains through training strategy adjustments without any architectural changes, which underscores the architecture’s continued efficacy. Through a small modification, they reinstated the performance of PointNet++ to a state-of-the-art level. This aligns with our approach, akin to the targeted adjustments in SOUL, suggesting the potential for significant performance improvements with subtle modifications.\n\nAnother significant advantage worth highlighting is PointNet++ has smaller latency, fewer parameters, and lower memory consumption (see Table 6 in Choe et al.[ 2]). Latency significantly influences training speed, and due to the use of rebalanced loss, SOUL demands a big number of training epochs. Hence, PointNet++’s advantage of lower latency proves highly advantageous for our purposes.\n\nIndeed, a comparison extended to other DL methods with different architectures would be feasible, but we just did not have the resource to try that.\n\n(1) Limitations & Weaknesses: The innovation part seems lacking.\n\nWe agree that our individual ingredients may be established techniques. However, combining them in a way resulting on satisfying results on such challenging data set, required a certain amount of expertise and design that resulted in a novel proposal. This is also confirmed by the absence of real competitors among methods addressing similar applications and datasets. \n\nAs mentioned in the global response and in the response to reviewers yADz and bCLj, the innovative parts of our proposal lie in (1) a whole pipeline for the forest segmentation task with potential community impact, (2) GVD preprocessing for sparse forest data and (3) rebalanced loss function designed for unbalanced ULS forest recordings. We will try to better emphasize these aspects in our revisions.\n\nReferences\n\n[1] G. Qian, Y. Li, H. Peng, J. Mai, H. A. A. K. Hammoud, M. Elhoseiny, and B. Ghanem, “Pointnext: Revisiting pointnet++ with improved training and scaling strategies,” 2022.\n\n[2] J. Choe, C. Park, F. Rameau, J. Park, and I. S. Kweon, “Pointmixer: Mlp-mixer for point cloud understanding,” 2022."}, {"review_id": "B1tdVagtOb", "reviewer": "Reviewer_81in", "summary": "They describe an approach for automatically segmenting a LIDAR scan of a forest into wood and leaf points.  They train a PointNet++ model and use resampling to address the extreme class imbalance in the data.  In comparison to previous methods they achieve a much higher balanced accuracy on their dataset.", "questions": "It was not clear to me why they needed to first cluster the tree into large segments; perhaps they could better explain the motivation for this.", "limitations": "Some limitations were discussed but there was not a section specifically labeled \"limitations.\"  Ethical implications were not discussed.", "rating": 3, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The approach they propose is novel to the best of my knowledge.  The presentation is fairly detailed and clear.  \n\nAccording to their experimental results (Table 1), they greatly outperform previous methods in terms of specificity and balanced accuracy.  ", "weaknesses": "There is probably not enough of a contribution here in terms of machine learning methods for this paper to be appropriate for NeurIPS.  The paper is rather narrow in scope and specific to the application of wood-leaf segmentation.  Furthermore, they combine existing techniques such as extraction of PCA features, PointNet++, and resampling to handle imbalanced data.  As an application study, I would see this type of work as more appropriate for an applied machine learning conference / journal or a forestry / ecology journal.\n\nAlso, PointNet++ has been used before for wood-leaf segmentation -- this paper was not cited:\n\nXi, Zhouxin, et al. \"See the forest and the trees: Effective machine and deep learning algorithms for wood filtering and tree species classification from terrestrial laser scanning.\" ISPRS Journal of Photogrammetry and Remote Sensing 168 (2020): 1-16.\n\nComments on presentation:\n* L149 need spaces in the vector notation [0 0 1] (you can do $[0 ~ 0 ~ 1]$ for example)\n* Eq. 7: if the voxels are adjacent, wouldn't the manhattan distance be 1 anyway?  How is \"adjacent\" defined?\n* Figure 3: the numbers in the color bar are too small to read\n* Eq. 9: the loss doesn't seem to be properly defined.  When y_k = 0, the loss term = 0?\n* L254: \"big trunk\" -> \"large trunks\"\n* Figure 5, 6: what do the colors mean here?  The statement \"any other approaches developed for dense point cloud are ineffective\" is too broad.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for very helpful comments, and thank you for bringing this recent work[1] to our attention.\nWe would add the discussion and cite the paper in the final revision.\n\nQ1: It was not clear to me why they needed to first cluster the tree into large segments; perhaps they\ncould better explain the motivation for this.\n\nThe ULS data often covers several hectares, or even hundreds of hectares. The situation in tropical\nforests is also highly complex, with various types and sizes of trees densely packed together. The\nsignificant memory demands make it nearly impossible to process all the data at once, leading us to\nadopt a spatial split schema approach as a necessary compromise.\n\nWe can select data randomly from the whole scene, but we agree that selecting data randomly can\nresult in a sparse and information-poor sample. Or we can employ a divide and conquer strategy to\nhandle the chaotic, big volume, and complex ULS data. That’s why we propose GVD, a method that\ninvolves breaking down the data into more manageable subsets (refer to Figure 1 in rebuttal_figures.pdf), allowing us to handle the intricacies\nand extract meaningful insights in a more systematic manner. This approach enables us to retain the\ninformation-rich aspects of the data while overcoming computational challenges associated with the\nsheer volume of information.\n\nIn our global response to all the reviewers, we have introduced an alternative spatial split scheme\ncalled the \"sliding window\". This method involves dividing the entire scene into multiple overlapping\ncuboids. Within each cuboid, we select training, validation, and testing samples. While this isn’t a\nstrict ablation study of GVD, but this allows us to observe the clear impact of border effects (refer to Figure 2 in rebuttal_figures.pdf) and the disruption of spatial information in point cloud data.\n\nResponse to weakness & comments:\n\n(1) weakness : There is probably not enough of a contribution here in terms of machine learning methods for this paper to be appropriate for NeurIPS. The paper is rather narrow in scope and specific to the application of wood-leaf segmentation.\n\nThe proposed rebalanced loss addresses the persistent challenge of class imbalance in real-world\ndata, which has practical implications in various domains. By presenting at NeurIPS, we aim to raise\nvisibility and engage with experts in the field. Additionally, our GVD preprocessing method offers a\nnew perspective for handling spatial dependencies. We firmly believe our contributions will benefit the research community, with potential applications beyond our immediate focus. Our paper is in line with \"Machine learning for sciences (e.g. climate, health, life sciences, physics, social sciences)\" of NeurIPS inviting topics, and we believe our work aligns well with the conference's interdisciplinary focus.\n\nPlus, as you mentioned, PointNet++ has been employed not only in the study[1] but also by\nothers[2]. However, none of these works have successfully applied this architecture to ULS data.\nFurthermore, we have undertaken targeted adjustments to the PointNet++ backbone for our specific\ncontext (see section B, Architecture of DL model, in our paper).\n\n(2) Eq. 7: if the voxels are adjacent, wouldn’t the Manhattan distance be 1 anyway? How is \"adjacent\"\ndefined?\n\nAdjacent voxels are defined as voxels that share a common surface, if the voxel a and voxel b are adjacent, their Manhattan distance is 1. Consider voxel b and voxel c are adjacent, while voxel a and voxel c are not. In this case, the Manhattan distance  (or geodesic distance in paper) between voxel a and voxel c is 2. What we want to calculate is the Manhattan distance between one of the lowest voxel, this voxel is fixed once it is chosen, and all the other voxels within the same component given by GVD. The voxel situated at the lowest position along the z-axis is called \"lowest voxel.\"\n\n(3) Eq. 9: the loss doesn’t seem to be properly defined. When yk = 0, the loss term = 0?\n\nWe will define it in a more proper way in the revised version.\n\n(4) Figure 5, 6: what do the colors mean here? The statement \"any other approaches developed for\ndense point cloud are ineffective\" is too broad.\n\nRed indicates high likelihood of being a wooden point, while blue indicates high likelihood of being\na leaf point. The color gradient can be referred to in Figure 6’s color bar. \n\nIndeed, we will revise the statement in the final version.\n\n(5) typos problem\n\nWe will correct all the typos accordingly in the revised version.\n\nReferences\n\n[1] Z. Xi, C. Hopkinson, S. B. Rood, and D. R. Peddle, “See the forest and the trees: Effective\nmachine and deep learning algorithms for wood filtering and tree species classification from\nterrestrial laser scanning,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 168,\npp. 1–16, 2020.\n\n[2] S. Krisanski, M. S. Taskhiri, S. Gonzalez Aracil, D. Herries, A. Muneri, M. B. Gurung, J. Mont-\ngomery, and P. Turner, “Forest structural complexity tool—an open source, fully-automated tool for measuring forest point clouds,” Remote Sensing, vol. 13, no. 22, 2021.\n\n"}, {"review_id": "JMMgFbuXTs", "reviewer": "Reviewer_yADz", "summary": "This paper proposes a dataset and an algorithm for 3D semantic segmentation in forest scenes. From data collection to the algorithm design, this paper covers the whole pipeline that are oriented for forest segmentation. In terms of the algorithmic part, the solution itself is not fully satisfied with me, but I enjoyed the geometric feature computation (Sec.3.1). Moreover in data pre-partitioning part, I understand the intention of such design and it makes sense to me.", "questions": "Please check the weakness section above. Though I wrote down not much contents on this review, I think that this paper is quite solid and reasonable. I really enjoyed reading this submission.", "limitations": "It's fine with me.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper covers the whole pipeline, from data collection to algorithm inference (semantic segmentation), for tree segmentation. While most of the 3D semantic segmentation methods, PointNet++, Point Transformer (ICCV 21), PointMixer (ECCV 22), PointNeXT (Neurips 2022) only deal with 3D semantic segmentation within the limited indoor scenes (S3DIS dataset or ScanNet dataset), this paper newly introduce the tree/forest segmentation using LiDAR point clouds.\n\nNot just an algorithm part, this paper also covers data collection and data preprocessing such that this paper covers the whole pipeline for the forest segmentation task.", "weaknesses": "Honestly, I could not find the weakness of this paper. Nonetheless, I have a minor question. Can you conduct an ablation study for the rebalance loss? If possible, I want to see the quantitative/qualitative result based on this loss design.\n\nExcept that question, I am fully okay with this paper.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your very positive and supportive review. We truly appreciate your acknowledgment of our efforts.\n\nQ1: Can you conduct an ablation study for the rebalance loss? If possible, I want to see the quantitative/qualitative result based on this loss design.\n\nSince the class imbalance problem is vital, we do early replace the cross entropy by focal loss, and we are certain that the cross entropy version will generally predict all the points as leaf. You can see the qualitative result of the focal loss version by referring to Figure 3 in rebuttal_figures.pdf. The qualitative results can be found in Table 1 of the rebuttal_figures.pdf."}, {"review_id": "LrcJxcCCsR", "reviewer": "Reviewer_bCLj", "summary": "The authors proposed a method for semantic segmentation applied to unmanned aerial vehicle (UAV) laser scanning (ULS), namely SOUL, to discriminate leaf from wood points. It is based on PointNet++ with an additional sampling scheme and an innovative training loss function to handle the high imbalance between the classes. The SOUL method relies on the coordinates of the points to increase its range of application to other forests and other sensors. It also includes 4 point-wise geometric features computed at 3 scales to characterize each point.\nThe geodesic voxelization decomposition (GVD) is also introduced as a preprocessing method to partition the ULS data while preserving the topology of the point cloud.\nExperiments have been conducted on a dataset recorded in a French Guiana tropical forest. The proposed method has reached the best results on 3 over 6 evaluation metrics, including metrics adapted to unbalanced datasets.\nThe approach has also been qualitatively tested on open source datasets recorded in Australia and Germany with open source datasets showing a potential generalization on other forests and other LiDAR sensors.", "questions": "Questions:\n\n1/ Will the dataset be release publicly? Even though the code is available, the only numerical results are presented on the ULS dataset in French Guiana. For reproducibility reasons (both training and testing), the final rate of this submission will be conditioned to the release of the dataset which is not mentioned in the paper. Note that it would be a noticeable contribution since it will be the only annotated ULS dataset for semantic segmentation.\n\n2/ Footnote 3, P7 \"MCC is a very good metric for evaluating binary classifier against class imbalance issue.\" What \"very good\" means  in this context?\n\n3/ It is not perfectly clear how the annotations of the French Guiana forest have been built. Do you have any additional comment on this topic?\n\n\nComments:\n\n1/ There is no reference made in the core text to the appendices, and vice versa, making the bridge difficult for the reader. These documents should not be considered independently, both should refer to each other.\n\n2/ A focus of qualitative and quantitative results at the limits of the branches would have been interesting to conduct since it concentrates most of the uncertainties. It would show that the model is not learning just a smart threshold on the z axis to discriminate the trunk from the leaves.\n\n3/ Please be more specific than \"partitioned\" (L227) to qualify a spatial split. This information is only clearly available in the video of the Appendices.\n\n4/ Typo: it would be nice to have the full name of the algorithm appearing in the title of Algorithm 1 in P5.\n\n5/ Table 1: if the top 1 results are in bold per columns, all top 1 results should be highlighted, included results from competing methods.\n\n6/ Typo: L290 \"Quantitative results demonstrated are shown in Figure 6.\" These are qualitative results.\n\n7/ Appendices L43 \"It is evident 43 that SOUL outperforms all other methods on ULS data and achieves state-of-the-art performance on 44 metrics such as Specificity, Balanced Accuracy and G-mean.\" It is not straightforward since the standard deviations for the competing methods have not been provided, neither in Table 2 nor in Table 3 (Appendices). It is not possible to evaluate if the variance of the other methods include the average performance of the proposed one. Note that the methodology used to create the intervals are not usual in this submission, but still relevant. A more common practice is to estimate the standard deviation of the performances of several models with different initializations of the network.\n", "limitations": "The limits of the presented method has been addressed by the authors. The potential negative societal impacts have not been mentioned (e.g. military applications, UAV surveillance). The release of the dataset has not been mentioned which is a major limitation since the proposed method has been designed and validated on it and the only numerical results are based on its annotations.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1/ The application of semantic segmentation to LiDAR recordings of forests is a high priority for climate change and global warming understanding and mitigation. This original work could have a potentially high impact for reforestation/afforestation monitoring and carbon stock estimation.\n\n2/ The proposed SOUL method is the first to be adapted to the density and unbalance of ULS forest recordings. The proposed GVD preprocessing method is also relevant for other LiDAR sensors and type of forests.\n\n3/ The authors have successfully performed experiments on a datasets recorded in a tropical forest in French Guiana, showing best results compared to a few competing methods and according to metrics specifically adapted to unbalanced datasets. Additional qualitative experiments conducted on unannotated datasets have shown a potential generalization on other forests and other LiDAR sensors.\n\n4/ The paper is well written and motivated with relevant arguments.", "weaknesses": "1/ There is a lack of related works and comparisons using other point cloud architectures which could have led to better performances [1, 2]. The selection of PointNet++ has been motivated by L127 \"the lower GPU requirements compared with transformer-based models developed in recent years\". The model should be selected as a trade off between the GPU consumption and the performances of different architecture. These experiments would have been appreciated to support the argument.\n\n2/ An ablation study of the proposed geometric features would have been appreciated. What is the impact of each proposed feature?  What is the actual impact of these features against using the raw point cloud? Is a model such as PointNet++ capable of estimating these geometric characteristics internally?\n\n3/ Even if the LiDAR point cloud is affected by atmospheric characteristics, it would have been interesting to see the performances of the proposed method using the reflectance as an additional feature per point.\n\n4/ The lack of experiments, in particular with competing methods, ablation studies and standard deviations (see next section), makes the submission questionable as to the significance of the results.\n\n[1] Y. Guo et al., Deep Learning for 3D Point Clouds: A Survey. In TPAMI 2020.\n\n[2] B. Fei et al., Comprehensive Review of Deep Learning-Based 3D Point Cloud Completion Processing and Analysis. In TITS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your acknowledgment of our effort and the very helpful comments.\n\nQ1: Will the dataset be release publicly? ...\n\nWe will release the labelled ULS data publicly along with the SOUL code, since this kind of data is\nstill extremely rare indeed. It should make a useful contribution to the community of experts working\non point cloud semantic segmentation of natural environments. \n\nQ2: For metric MCC, what \"very good\" means in this context?\n\nMCC = (TP * TN - FP * FN) / sqrt((TP + FP) * (TP + FN) * (TN + FP) * (TN + FN))\n\nFor binary classification, the calculation of the Matthews correlation coefficient (MCC) metric uses\nall the four quantities (TP, TN, FP and FN) of the confusion matrix, and produces a high score only if\nthe prediction obtained good results in all the four confusion matrix quantities[1][ 2], proportionally\nboth to the size of positive elements and the size of negative elements in the dataset. (see section Conclusions in Davide\nChicco and Giuseppe Jurman [1]).\n\nThis is exactly what we need in our context.\n\nQ3: It is not perfectly clear how the annotations of the French Guiana forest have been built. Do you\nhave any additional comment on this topic?\n\nAnnotation was done on high-resolution Terrestrial Laser Scanning point cloud acquired simultaneously. The labels were subsequently transferred to the less dense ULS point cloud. The procedure used to annotate the TLS point cloud follows the description given in Martin-Ducup et al.[ 3] under their section entitled \"Human assisted generation of control data\".\n\nResponse to weakness:\n\n(1) There is a lack of related works ...\n\nThe Table 6 in Choe et al.[4 ] give such pertinent information. PointNet++ exhibits advantages in\nterms of memory consumption and training speed, with the results being relatively comparable. While\nI am inclined to conduct further experiments about the GPU consumption and the performances of\ndifferent architecture, but resource constraints currently limit our capacity to do so.\n\nIn essence, transformer-based architecture will do a lot of dot products to compute a sort of similarity\nscore between the query and key vectors. This nature inherently leads to slower processing speeds.\n\n(2) An ablation study of the proposed geometric features ...\n\nWe can share the results of an experiment which compares the full model with downgraded model\nversions where only a single geometric feature was used at a single scale (see Figure 4 and Table 1 in rebuttal_figures.pdf). Those results clearly show the benefits of including precomputed multiple geometric features at multiple scales. We think linearity and verticality contribute more to trunk recognition, whereas PCA1 and Sphericity are more efficient inside canopy. \n\nPointNet++ is able to learn more sophisticated features, but for empirically chosen geometric attributes, it may be not capable to learn identical ones.\n\n(3) ...using the reflectance as an additional feature per point.\n\nIndeed, reflectance may in some circumstances help discriminate leaf from wood and was used for instance in Wu et al.[5]. However, this makes the model sensor specific as different wavelengths are commonly used in different sensors.\n\nThe main reason we decided not to use reflectance-derived information is its high variability and\nlack of specificity. In fact, reflectance cannot discriminate leaf from wood reliably at 905nm the\nwavelength at which our sensor operated (see for instance Figure 2 in Brede et al. [6]). In addition,\nmultiple returns (pulse fragmentation), unknown object orientation, and surface wetness all contribute\nto adding noise to the apparent reflectance associated with each return(for a discussion of that point\nplease see for instance Vincent et al. [7]).\n\n(4) The lack of experiments with competing methods...\n\nWhile we only compare our method to two others, the latter are commonly recognized as the best\nalternatives in the context we are dealing with (semantic segmentation of forest point clouds). A\ncomparison extended to other DL methods with different architectures, which have no proven record\nof performance for the targeted task would be feasible, but we just did not have the resource to try\nthat.\n\nResponse to comments:\n\nWe will correct all the typos accordingly in the revised version, and thanks for the practice, we will use this common practice to estimate sd in the future.\n\nReferences\n\n[1] D. Chicco and G. Jurman, “The advantages of the matthews correlation coefficient (mcc) over f1\nscore and accuracy in binary classification evaluation,” BMC Genomics, vol. 21, 01 2020.\n\n[2] J. Yao and M. Shepperd, “Assessing software defection prediction performance,” in Proceedings\nof the Evaluation and Assessment in Software Engineering, ACM, apr 2020.\n\n[3] O. Martin-Ducup, I. Mofack, Gislain, D. Wang, P. Raumonen, P. Ploton, B. Sonké, N. Barbier,\nP. Couteron, and R. Pélissier, “Evaluation of automated pipelines for tree and plot metric\nestimation from tls data in tropical forest areas,” Annals of Botany, vol. 128, pp. 753–766, 04\n2021.\n\n[4] J. Choe, C. Park, F. Rameau, J. Park, and I. S. Kweon, “Pointmixer: Mlp-mixer for point cloud\nunderstanding,” 2022.\n\n[5] B. Wu, G. Zheng, and Y. Chen, “An improved convolution neural network-based model for\nclassifying foliage and woody components from terrestrial laser scanning data,” Remote Sensing,\nvol. 12, no. 6, 2020.\n\n[6] B. Brede, H. M. Bartholomeus, N. Barbier, F. Pimont, G. Vincent, and M. Herold, “Peering\nthrough the thicket: Effects of uav lidar scanner settings and flight planning on canopy volume\ndiscovery,” International Journal of Applied Earth Observation and Geoinformation, vol. 114,\np. 103056, 2022.\n\n[7] G. Vincent, P. Verley, B. Brede, G. Delaitre, E. Maurent, J. Ball, I. Clocher, and N. Barbier,\n“Multi-sensor airborne lidar requires intercalibration for consistent estimation of light attenuation\nand plant area density,” Remote Sensing of Environment, vol. 286, p. 113442, 2023.\n\n"}]

Author rebuttal:
We thank the reviewers for their comments and for underlining the existence of new and good
features in our work, in particular regarding new methodological contributions, reproducibility and
relevance of the experiments, while also suggesting improvement and clarification. We also note that
the importance of forests segmentation, as an essential step to climate change and global warming
understanding and mitigation, has been acknowledged. We summarize below the main points addressed in our response:

(1) GVD ablation study:

Further elaboration on the benefits of the GVD method is warranted in our paper. The ULS data often covers several hectares, or even hundreds of hectares. The situation in tropical forests is also highly complex, with various types and sizes of trees densely packed together. The significant memory demands make it nearly impossible to process all the data at once, leading us to adopt a spatial split scheme approach as a necessary compromise.

We can select data randomly from the whole scene, but selecting data randomly can result in a sparse and information-poor sample. An alternative is to employ a divide and conquer strategy to handle the chaotic, big volume, and complex ULS data. That's why we propose GVD, a method that involves breaking down the data into more manageable subsets (refer to Figure 1 in rebuttal_figures.pdf), allowing us to handle the intricacies and extract meaningful insights in a more systematic manner. This approach enables us to retain the information-rich aspects of the data while overcoming computational challenges associated with the sheer volume of data.

Prior to employing the GVD method, we initially adopted a more intuitive approach. The data was partitioned in unit of voxel, serving as component for batch preparation through down-sampling. However, this approach gives rise to border effects, particularly impeding SOUL's focus on the meticulous segmentation of intricate branch and leaf within tree canopy. The segmentation of cubes led to the emergence of noise point cluster along the voxel edges. ULS have more recording on tree canopy, so the presence of noise point cluster on voxel edges is more severe on tree canopy, which imposes bigger obstacles to achieving our goal.

We experimented with cuboids, a choice aimed at preserving greater semantic information within each component sample and expanding the spatial range for batch selection. Similar to a "sliding window", we can systematically traverse the entire forest with overlapping coverage in this way. But border effects persisted (see Figure 2 in rebuttal\_figures.pdf), prompting the introduction of the GVD method, which led to a substantial improvement. A comparison between the result of the downgraded version using sliding window as spatial split schema and the individual-component point performance of full version SOUL is illustrated (see Figure 1 in rebuttal\_figures.pdf). Notably, in component N°552, SOUL effectively discriminate trunk points from leaf points that were previously entirely intertwined, surpassing our initial expectations. So, in alignment with the observations of Reviewer bCLj, we hold the conviction that the proposed GVD preprocessing method is relevant beyond the confines of the present study, extending to various LiDAR sensors and diverse forest types.

A new video (3D version of Figure 2 in rebuttal_figures.pdf) demonstrating the use of cuboid as a spatial split method has been provided to AC.

(2) Novelty:

Our main contribution lies in the complete handling of the full pipeline from data
collection to final segmentation. To our knowledge, this is quite unique in forestry applications,
where either datasets are limited to much simpler structures, or analysis techniques show limited
performance. In this work, we combine advanced and original deep learning methods and principles
to get what is a first baseline on segmentation of ULS LiDAR data. Plus, our model SOUL exclusively depends on coordinates and does not incorporate RGB information, which is different from indoor scenes or autopilot contexts.

(3) Relevance to NeurIPS:

Our work makes use and proposes new developments and contributions in
core deep learning techniques. Although it focuses on point cloud processing, we believe that some
of the principles and ideas are transferable to other data structures and problems. As regards the
target application, according to this year call for papers, the conference scope includes explicitly life
and natural sciences. In addition, in contrast to some of the reviewers, we believe the topic of our
work may be of interest to the NeurIPS readership, especially as mentioned above, for its potential high
environmental applications impact, which is a rising concern for a lot of junior and senior researchers.
In addition, to foster collaboration and facilitate meaningful exchange within the research community,
we have made the decision to openly share our code and data, hoping to engage with fellow researchers
in a more inclusive and productive manner on these important applications.

(4) The video provided before:

Just in case, a video hyperlink has been embedded within the closing word of the first paragraph in the supplementary. This video shows the qualitative result of SOUL model and some comparisons between different methods.

Meta-review: This paper introduces a dataset and methods for segmentation of sparse point clouds from UAVs for monitoring vegetation. The problem is urgently relevant and a good match for machine learning, and poses unique challenges. The biggest concern raised was in "novelty", but I concur with several of the reviewers in seeing this paper as an effective example of application-guided innovation, where the methodological advances were appropriately motivated by application-specific challenges and criteria for success. Given the utility of the dataset to the machine learning community and the solid baseline methods, I recommend the paper for acceptance. However, I do believe that adding as many comparison methods and ablations as possible in the camera-ready will improve the usefulness to the NeurIPS community.
Decision: Accept (poster)