SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yuchen Bai $^{1}$  Jean-Baptiste Durand $^{2*}$  Grégoire Vincent $^{2}$  Florence Forbes $^{1}$
<sup>1</sup>Univ. Grenoble Alpes, CNRS, Inria, Grenoble INP, LJK, Grenoble, France  
<sup>2</sup>AMAP, Univ. Montpellier, CIRAD, CNRS, INRAE, IRD, Montpellier, France {yuchen.bai,florence.forbes}@inria.fr  
jean-baptiste.durand@cirad.fr  
gregoire.vincent@ird.fr
LiDAR (Light Detection And Ranging) has become an essential part of the remote sensing toolbox used for biosphere monitoring. In particular, LiDAR provides the opportunity to map forest leaf area with unprecedented accuracy, while leaf area has remained an important source of uncertainty affecting models of gas exchanges between the vegetation and the atmosphere. Unmanned Aerial Vehicles (UAV) are easy to mobilize and therefore allow frequent revisits, so as to track the response of vegetation to climate change. However, miniature sensors embarked on UAVs usually provide point clouds of limited density, which are further affected by a strong decrease in density from top to bottom of the canopy due to progressively stronger occlusion. In such a context, discriminating leaf points from wood points presents a significant challenge due in particular to strong class imbalance and spatially irregular sampling intensity. Here we introduce a neural network model based on the Pointnet ++ architecture which makes use of point geometry only (excluding any spectral information). To cope with local data sparsity, we propose an innovative sampling scheme which strives to preserve local important geometric information. We also propose a loss function adapted to the severe class imbalance. We show that our model outperforms state-of-the-art alternatives on UAV point clouds. We discuss future possible improvements, particularly regarding much denser point clouds acquired from below the canopy.
In the past decades, LiDAR technology has been frequently used to acquire massive 3D data in the field of forest inventory (Vincent et al. [1]; Ullrich & Pfennigbauer [2]). The acquisition of point cloud data by employing LiDAR technology is referred to as laser scanning. The collected point cloud data provides rich details on canopy structure, allowing us to calculate a key variable, leaf area, which controls water efflux and carbon influx. Monitoring leaf area should help in better understanding processes underlying flux seasonality in tropical forests, and is expected to enhance the precision of climate models for predicting the effects of global warming. There are various types of vehicles for data collection, with ground-based equipment and aircraft being the most commonly employed. The former operates a bottom-up scanning called terrestrial laser scanning (TLS), providing highly detailed and accurate 3D data. Scans are often acquired in a grid pattern every  $10\mathrm{m}$  and co-registered into a single point cloud. However, TLS requires human intervention within the forest, which is laborious and limits its extensive implementation. Conversely, airborne laser scanning (ALS) is much faster and can cover much larger areas. Nonetheless, the achieved point density is typically two
orders of magnitude smaller due to the combined effect of high flight altitude and fast movement of the sensor. Additionally, occlusions caused by the upper tree canopy make it more difficult to observe the understory vegetation.
In recent years, the development of drone technology and the decreasing cost have led to UAV laser scanning (ULS) becoming one favored option (Brede et al. [3]). It does not require in-situ intervention and each flight can be programmed to cover a few hectares. The acquired data is much denser than ALS (see Figure 1(a) and Figure 1(b)), which provides us with more comprehensive spatial information. Increasing the flight line overlap results in multiple angular sampling, higher point density and mitigates occlusions. Although the data density is still relatively low, compared with TLS, ULS can provide previously unseen overstory details due to the top-down view and overlap flight line. Furthermore, ULS is considered to be more suitable for conducting long-term monitoring of forests than TLS, as it allows predefined flight plans with minimal operator involvement.
Consequently, leaf-wood semantic segmentation in ULS data is required to accurately monitor foliage density variation over space and time. Changes in forest foliage density are indicative of forest functioning and their tracking may have multiple applications for carbon sequestration prediction, forest disease monitoring and harvest planning. Fulfilling these requirements necessitates the development of a robust algorithm that is capable to classify leaf and wood in forest environments. While numerous methods have demonstrated effective results on TLS data (see Section 2), these methods cannot be applied directly to ULS, due in particular to the class imbalance issue: leaf points account for about  $95\%$  of the data. Another problem is that many methods rely on the extra information provided by LiDAR devices, such as intensity. In the context of forest monitoring, intensity is not reliable due to frequent pulse fragmentation and variability in natural surface reflectivity (see Vincent et al. [5]). Furthermore, the reflectivity of the vegetation is itself affected by diurnal or seasonal changes in physical conditions, such as water content or leaf orientation (Brede et al. [3]). Therefore, methods relying on intensity information (Wu et al. [6]) may exhibit substantial
(a) ALS
Figure 1: Point clouds produced by three scanning modes on the same area  $(20\mathrm{m}\times 20\mathrm{m}\times 42\mathrm{m})$  , illustrate how much the visibility of the understory differs. The colors in the figure correspond to different labels assigned to the points, where red and green indicate leaves and wood, respectively. Blue points are unprocessed, so labeled as unknown. TLS data were initially subjected to semantic segmentation using LeWos [4] and subsequently manually corrected. Next, the K-nearest neighbors (KNN) algorithm was used to assign labels to ALS and ULS data based on the majority label among their five nearest neighbors in TLS data.
(b) ULS
(c) TLS
variations in performance across different locations and even within the same location for different acquisition batches. To address this issue, certain methods (LeWos proposed by Wang et al. [4]; Morel et al. [7]) have good results while exclusively utilizing the spatial coordinates of LiDAR data.
Inspired by the existing methods, we propose a novel end-to-end approach SOUL (Semantic segmentation On ULs) based on PointNet++ proposed by Qi et al. [8] to perform semantic segmentation on ULS data. SOUL uses only point coordinates as input, aiming to be applicable to point clouds collected in forests from various locations worldwide and with sensors operating at different wavelengths. The foremost concern to be tackled is the acquisition of labeled ULS data. Since no such data set existed up to now, we gathered a ULS data set comprising 282 trees labeled as shown in Figure 4. This was achieved through semi-automatic segmentation of a coincident TLS point cloud and wood/leaf label transfer to ULS point cloud. Secondly, the complex nature of tropical forests necessitates the adoption of a data pre-partitioning scheme. While certain methods (Krisanski et al. [9]; Wu et al. [10]) employ coarse voxels with overlap, such an approach leads to a fragmented representation and incomplete preservation of the underlying geometric information. The heterogeneous distribution of points within each voxel, including points from different trees and clusters at voxel boundaries, poses difficulties for data standardization. We introduce a novel data preprocessing methodology named geodesic voxelization decomposition (GVD), which leverages geodesic distance as a metric for partitioning the forest data into components and uses the topological features, like intrinsic-extrinsic
ratio (IER) (He et al. [11]; Liu et al. [12]), to preserve the underlying geometric features at component level (see Section 1). The last issue concerns the class imbalance problem during the training stage. To address this issue, we developed a novel loss function named the rebalanced loss, which yielded improved performance compared with the focal loss (Lin et al. [13]) for our specific task. This enhancement resulted in a  $23\%$  increase in the ability to recognize wood points, see Table 1.
The contribution of our work is three-fold. First, SOUL is the first approach developed to tackle the challenge of semantic segmentation on tropical forest ULS point clouds. SOUL demonstrates better wood point classification in complex tropical forest environments while exclusively utilizing point coordinates as input. Experiments show that SOUL exhibits promising generalization capabilities, achieving good performance even on data sets from other LiDAR devices, with a particular emphasis on overstory. Secondly, we propose a novel data preprocessing method, GVD, used to pre-partition data and address the difficult challenge of training neural networks from sparse point clouds in tropical forest environments. Third, we mitigate the issue of imbalanced classes by proposing a new loss function, referred to as rebalanced loss function, which is easy to use and can work as a plug-and-play for various deep learning architectures. The data set (Bai et al. [14]) used in the article is already available in open access at https://zenodo.org/record/8398853 and our code is available at https://github.com/Na1an/phd_mission.
Classical methods. Classical methods are prevalent for processing point cloud data in forest environments, especially for semantic segmentation of TLS data, where they have achieved high levels of efficacy and applicability in practice. One such method, LeWos (Wang et al. [4]) uses geometric features and clustering algorithms, while an alternative method proposed by the first author of LeWos (Wang [15]) employs superpoint graphs (Landrieu & Simonovsky [16]). Moorthy et al. [17] use random forests (RF), and Zheng et al. [18] explore Gaussian mixture models. Other approaches, such as those proposed by Lalonde et al. [19], Itakura et al. [20], Vicari et al. [21], and Wan et al. [22], have also demonstrated effective semantic segmentation of TLS data.
Deep Learning methods. Pioneering works like FSCT proposed by Krisanski et al. [9] and the method proposed by Morel et al. [7] have shown promising results by using PointNet++ [8]. Shen et al. [23] use PointCNN (Li et al. [24]) as the backbone model to solve the problem of segmenting tree point clouds in planted trees. Wu et al. [6] propose FWCNN, which incorporates intensity information and provides three new geometric features. Windrim & Bryson [10] propose a method that combines Faster R-CNN [25] and Pointnet [26], which utilizes the bird's eye view technique for single tree extraction and performs semantic segmentation on individual trees. In general, deep neural network-based approaches have demonstrated better performance in semantic segmentation of TLS point clouds compared to classical methods.
Figure 2: Overview of SOUL. (a) We use only the coordinates of raw LiDAR data as input. (b) Four geometric features linearity, sphericity, verticality, and PCA1 are calculated at three scales using eigenvalues, then standardized. (c) GVD proposes partitioned components and performs data normalization within these components. (d) Training deep neural network. (e) Finally, output are point-wise predictions.
3 SOUL: Semantic segmentation on ULS
SOUL is based on PointNet++ Multi-Scale Grouping (MSG) [8] with some adaptations. The selection of PointNet++ is not only because of its demonstrated performance in similar tasks (Krisanski et al. [9]; Morel et al. [7]; Windrim & Bryson[10]), but also because of the lower GPU requirements (Choe et al. [27]) compared with transformer-based models developed in recent years, like the method proposed by Zhao et al. [28]. The main idea of SOUL lies in leveraging a geometric approach to extract preliminary features from the raw point cloud, these features are then combined with normalized coordinates into a deep neural network to obtain more abstract features in some high dimensional space [26]. We will introduce our method in detail as follows.
3.1 Geometric feature computation
At this stage, we introduce four point-wise features: linearity, sphericity, verticality, and PCA1, which are computed at multiple scales of  $0.3\mathrm{m}$ ,  $0.6\mathrm{m}$ , and  $0.9\mathrm{m}$  in this task. To calculate these features, we need to construct the covariance matrix  $\Sigma_{p_i}$  for each point  $p_i$ , compute its eigenvalues  $\lambda_{1,i} > \lambda_{2,i} > \lambda_{3,i} > 0$  and the corresponding normalized eigenvectors  $e_{1,i}, e_{2,i}, e_{3,i}$ . The local neighbors  $\mathcal{N}_{p_i}$  of  $p_i$  are given by:
$$
\mathcal {N} _ {p _ {i}} = \{q \mid q \in \mathcal {P}, d _ {e} (p _ {i}, q) \leq r \} \tag {1}
$$
where  $\mathcal{P}$  is the set of all points,  $d_{e}$  is the Euclidean distance and  $r\in \{0.3,0.6,0.9\}$ .
The covariance matrix  $\Sigma_{p_i}$  is:
$$
\Sigma_ {p _ {i}} = \frac {1}{\left| \mathcal {N} _ {p _ {i}} \right|} \sum_ {p _ {j} \in \mathcal {N} _ {p _ {i}}} (p _ {j} - \bar {p}) (p _ {j} - \bar {p}) ^ {T} \tag {2}
$$
where  $\bar{p}$  is the barycenter of  $\mathcal{N}_{p_i}$ .
The equations for computing the four characteristics are as follows:
Linearity  $L_{p_i}$  serves as an indicator for the presence of a linear 1D structure (Weinmann et al. [29]).
$$
L _ {p _ {i}} = \frac {\lambda_ {1 , i} - \lambda_ {2 , i}}{\lambda_ {3 , i}} \tag {3}
$$
Sphericity  $S_{p_i}$  provides information regarding the presence of a volumetric 3D structure (Weinmann et al. [29]).
$$
S _ {p _ {i}} = \frac {\lambda_ {3 , i}}{\lambda_ {1 , i}} \tag {4}
$$
Verticality  $V_{p_i}$  indicates the property of being perpendicular to the horizon (Hackel et al. [30]), LeWos [4] uses it as a crucial feature because of its high sensitivity to the tree trunk. Here  $[0 0 1]$  is the vector of the canonical basis.
$$
V _ {p _ {i}} = 1 - \left| \right. \langle [ 0 0 1 ], e _ {3, i} \left. \right\rangle\left. \right| \tag {5}
$$
PCA1  $PCA1_{p_i}$  reflects the slenderness of  $\mathcal{N}_{p_i}$ . The direction of the first eigenvector  $e_1$  is basically the most elongated in  $\mathcal{N}_{p_i}$ .
$$
P C A 1 _ {p _ {i}} = \frac {\lambda_ {1 , i}}{\lambda_ {1 , i} + \lambda_ {2 , i} + \lambda_ {3 , i}} \tag {6}
$$
Each point is endowed now with 12 features encompassing 3 scales, enabling a precise depiction of its representation within the local structure. These features are subsequently standardized and integrated into the model's input as an extension to point coordinates for the deep learning model. Refer to Supplementary Section E for single-scale and multi-scale ablation studies.
Algorithm 1 Geodesic Voxelization Decomposition (GVD)
Input: Voxel grid V; threshold τ on dgv; threshold γ on IER
Output: Voxel partition in I components C = {Ci, i = 1: I}
1: i ← 1	▷ Component id
2: function GVD(V, τ, γ)
3: while V ≠ {} do
4: Ci ← ∅, ier ← 0, gd ← 0
5: Choose v ∈ V
6: while ier < γ and gd < τ do
7: Choose v ∈ C(v)	▷ C(v) arises from BFS on v
8: gd ← dgv(v, v)
9: ier ← IER(v, v)	▷ Eq. (8)
10: Ci ← Ci ∪ {v}
11: end while
12: C = C ∪ Ci
13: i ← i + 1
14: end while
15: Return C
16: end function
3.2 Data pre-partitioning
The GVD algorithm 1 is a spatial split scheme used to partition the ULS data while preserving the topology of the point cloud. This approach enables the extraction of a set of representative training samples from raw forest data, while preserving the local geometry information in its entirety. The point cloud data is first voxelized into a voxel grid  $\mathcal{V}$  with a voxel size  $s$  equal to  $0.6\mathrm{m}$ , an empirical value, and equipped with a neighborhood system  $\mathcal{C}$ . Each voxel  $v$  is considered as occupied if it contains at least one point. The geodesic-voxelization distance  $d_{gv}$  between two voxels is defined as 1 if they are adjacent, and as the Manhattan distance between  $v$  and  $v^{\prime}$  in voxel space otherwise. This can be expressed mathematically as:
$$
d _ {g v} (v, v ^ {\prime}) = \left\{ \begin{array}{l l} 1, & \text {i f} v \text {a n d} v ^ {\prime} \text {a r e a d j a c e n t i n} \mathcal {C} \\ | x - x ^ {\prime} | + | y - y ^ {\prime} | + | z - z ^ {\prime} |, & \text {o t h e r w i s e} \end{array} \right. \tag {7}
$$
where  $(x,y,z)$  and  $(x',y',z')$  denote the integers coordinates referring to the index of  $v$  and  $v'$  respectively. We can establish an inter-voxel connectivity information with  $d_{gv}$  that enables the computation of the intrinsic-extrinsic ratio (IER):
$$
I E R (v, v ^ {\prime}) = \frac {d _ {g v} (v , v ^ {\prime})}{d _ {e} (v , v ^ {\prime})}. \tag {8}
$$
We believe that  $d_{gv}$  and  $IER$  may provide strong cues for the subsequent segmentation task.
Subsequently, we choose one lowest voxel  $\hat{v} \in \mathcal{V}$  on the vertical  $z$ -axis then employ the Breadth-first search (BFS) algorithm to explore its neighbors and the neighbors of neighbors  $\mathcal{C}(\hat{v})$  and calculate the corresponding  $d_{gv}(\hat{v}, \bar{v})$  and  $IER(\hat{v}, \bar{v})$  where  $\bar{v} \in \mathcal{C}(\hat{v})$ . This procedure terminates when the geodesic-voxelization distance  $d_{gv}$  exceeds  $\tau$  or the  $IER$  exceeds  $\gamma$ .  $\tau$  is set to 10 and  $\gamma$  is set to 1.5 in this task. All the points within  $\mathcal{C}(\hat{v})$  are extracted and consolidated as a component  $\mathbf{C}_i$  of the point clouds (see Figure 3(a) and Algorithm 1). The whole procedure is repeated until all the data is processed.
As the data is partitioned into multiple components, the points within one component  $\mathbf{C}_i$  are further sorted based on their geodesic-voxelization distance. In this arrangement, points inside one voxel  $v$  have the same geodesic distance  $d_{gv}(\hat{v}, v)$ , where  $\hat{v}$  is the lowest voxel inside  $\mathbf{C}_i$  (see Figure 3(b)). Thus, the GVD algorithm concludes at this stage. Moreover, minimum voxel number and minimum point number within a component are configurable, more details are provided in Supplementary Section C.
3.3 Normalization inside component
Shifting and scaling the point cloud coordinates before training a deep neural network is a widely adopted practice that can lead to improved training stability, generalization, and optimization efficiency, see Qi et al. [26]. Here, prior to training the model, the point coordinates are shifted to  $(0,0,0)$  and normalized by the longest axis among the three, all the point coordinates being confined within the  $(0,1)$  range. This prevents certain coordinates from dominating the learning process simply because they have larger values.
3.4 Deep neural network training
We employ a modified version of PointNet++ MSG for SOUL [8]. The data combination of points and features within a component is further partitioned into batches of 3,000, which serve as the input for the network. If a batch within a component has fewer than 3,000 points, the remaining points are randomly chosen from the same component. The labels of the leaves and wood points are 0 and 1, respectively. Define  $B_{k}$  as such a batch,  $|B_{k}| = 3,000$  and  $B_{k} = B_{k,0} + B_{k,1}$  where  $B_{k,0}$  and  $B_{k,1}$  respectively represent the disjoint sub-batches in  $B_{k}$  with ground-truth leaf points and w
(a) Partitioned trunk
Figure 3: (a) displays the components found by the GVD algorithm. (b) displays the geodesic distance within the corresponding component. The figures are illustrative, actual ULS data lacks such depicted complete tree trunks.
(b) Geodesic distance
Addressing class imbalance issue. Within the labeled ULS training data set, only  $4.4\%$  are wood points. The model is overwhelmed by the predominant features of leaves. Therefore, we propose a rebalanced loss  $L_{R}$  that changes the ratio of data participating to 1:1 at the loss calculation stage by randomly selecting a number of leaf points equal to the number of wood points.
In practice, the rebalanced loss is:
$$
L _ {R} \left(Y _ {B _ {k}}\right) = - \sum y _ {k} \log \left(\hat {p} _ {k}\right) + \left(1 - y _ {k}\right) \log \left(1 - \hat {p} _ {k}\right), y _ {k} \in \left(B _ {k, 0} ^ {\prime} \cup B _ {k, 1}\right). \tag {9}
$$
where  $Y_{B_k}$  specifies the ground truth labels of batch  $B_k$ ,  $\hat{p} \in [0,1]$  is the model-estimated probability for the label  $y = 1$  and  $B_{k,0}'$  is defined as
$$
B _ {k, 0} ^ {\prime} = \left\{ \begin{array}{l l} \text {d o w n s a m p l i n g} \left(B _ {k, 0}, \left| B _ {k, 1} \right|\right), & \text {i f} \left| B _ {k, 0} \right| \geq \left| B _ {k, 1} \right| \\ B _ {k, 0}, & \text {o t h e r w i s e .} \end{array} \right. \tag {10}
$$
where the downsampling procedure consists of randomly and uniformly selecting a subset of size  $|B_{k,1}|$  within  $B_{k,0}$  without replacement.
The ablation study for the loss function can be found in Supplementary Section D.
We first provide more details about various LiDAR data present in each data set. Secondly, the data preprocessing procedure is elaborated upon. Next, we delve into the specific configurations of model hyperparameters during the training process, along with the adaptations made to the PointNet++ architecture. Following that, we showcase the performance of SOUL on the labeled ULS data set. Finally, we demonstrate the generalization ability of SOUL on the other data sets.
4.1 Data Characteristics
The ULS data for this study were collected at the Research Station of Paracou in French Guiana  $(\mathrm{N}5^{\circ}18^{\prime}\mathrm{W}52^{\circ}53^{\prime})$ . This tropical forest is situated in the lowlands of the Guiana Shield, the average height of the canopy is  $27.2\mathrm{m}$  with top heights up to  $44.8\mathrm{m}$  (Brede et al. [3]). Four flights of ULS data were collected at various times over the same site, with each flight covering  $14,000~\mathrm{m}^2$  of land and consisting of approximately 10 million points, yielding a density of around  $1,000~\mathrm{pts / m^2}$ . In comparison, the point density of the labeled TLS data from the same site is about  $320,000~\mathrm{pts / m^2}$ . Further information pertaining to LiDAR scanners can be found in Supplementary Section B.
In addition, we used two public data sets without labels only for exploring the model's generalization ability. One is a ULS data set from Sandhausen, Germany (Weiser et al. [31]), the
Figure 4: The labeled ULS data set in French Guiana, where only  $4.4\%$  of the points correspond to wood. This significant class imbalance of leaf & wood presents a considerable challenge in discriminating wood points from ULS forest point cloud.
other one is a mobile laser scanning (MLS)  $^{3}$  data set from New South Wales (NSW), Australia (NSW & Gonzalez [32]). The Sandhausen data set uses the same equipment as we do, but exhibits a lower point density of about 160 pts/m $^{2}$ . The point density of the NSW data set is approximately 53,000 pts/m $^{2}$ .
The first step of the routine involves data cleaning for TLS and ULS. The points exhibiting an intensity lower than  $-20$  dB and those displaying high deviations (Pfennigbauer & Ullrich [33]) are removed, subsequently the TLS data are subsampled by setting a  $0.001\mathrm{m}$  coordinate precision. The next stage is to separate ground by applying a cloth simulation filter (CSF) proposed by Zhang et al. [34], which mimics the physical phenomenon of a virtual cloth draped over an inverted point cloud. The ULS data were subjected to a similar preprocessing pipeline, except that CSF on ULS (or ALS) performs poorly and was not used. Here we used the software TerraSolid ground detection algorithm on ALS and used the common ground model for all point clouds of the same plot.
4.3 Data annotation