SlowGuess/NeurIPS_2023_OCR huggingface dataset:
A subset of TLS trees with large trunk is labeled using LeWos algorithm (Wang et al. [4]), followed by a manual refinement process (Martin-Ducup et al. [35]) to improve the outcome accuracy. Then the k-nearest neighbors algorithm (KNN) is employed to assign the tree identifiers (treeIDs) and the labels of TLS points to ULS points. Specifically, for each point in ULS data set, the KNN algorithm selects five nearest points in the TLS data set as reference points and subsequently determines the corresponding label based on the majority consensus of the reference points. It is noteworthy that within TLS data, there exist three distinct label types: unknown, leaf and wood, as demonstrated in Figure 1(c), which respectively correspond to labels: -1, 0, and 1. The unknown label is also transferred to ULS points, then such points are filtered out for the next step. Unlabeled and therefore excluded points represent  $65\%$  of the ULS point cloud.
To facilitate model training and quantitative analysis, the labeled ULS data was partitioned into training, validation, and test sets based on treeID, with 221 trees in the training data set, 20 in the validation data set, and 40 in the test data set. This partitioning is visually demonstrated in a video, where the trees are distinctly grouped together within each data set.
2Points per square meter.
3The LiDAR device is integrated into a backpack, enabling a similar scanning process akin to TLS.
4.4 Implementation details
The Adam optimization algorithm is chosen as the optimizer and the learning rate is 1e-7. Following the practice proposed by Smith et al. [36], we employ a strategy of gradually increasing the batch size by a factor of two approximately every 1,000 epochs until reaching a final batch size of 128 instead of decreasing the learning rate during the training process. According to our experience, achieving good results requires a substantial duration, typically exceeding 3,000 epochs. In this paper, the prediction results are obtained from a checkpoint saved at the 3161st epoch. At this epoch, SOUL achieved a Matthews Correlation Coefficient (MCC) value of 0.605 and an AUROC value of 0.888 on the validation data set.
The MCC expression is:
$$
\mathbf {M C C} = \frac {T P \times T N - F P \times F N}{\sqrt {(T P + F P) (T P + F N) (T N + F P) (T N + F N)}} \tag {11}
$$
For binary classification, the calculation of the MCC metric uses all four quantities (TP, TN, FP and FN)4 of the confusion matrix, and produces a high score only if the prediction is good for all quantities (see Yao & Shepperd [37]), proportionally both to the size of positive elements and the size of negative elements in the data set (Chicco et al. [38]).
In our task, an output in the form of a probability is preferred, as it is more consistent with the data collection process, since the LiDAR footprint yielding each single echo is likely to cover both wood and leaf elements. However, to facilitate performance comparison with other methods, the model generates a discrete outcome that assigns a label to each point. If the probability of a point being classified as a leaf exceeds the probability of being classified as wood, it is labeled as a leaf and vice versa. Additionally, the probabilistic outputs indicating the likelihood of being a leaf or wood point are retained. More details are provided in the Supplementary Section B.
4.5 Results on French Guiana ULS data
Comparatively to the prevailing methods employed for forest point clouds, our SOUL approach improves semantic segmentation on ULS forest data by large margins. The results are summarized in Table 1, while Figure 5 highlights the performance within the tree canopies. In terms of quantitative analysis, SOUL demonstrates a specificity metric of  $63\%$  in discerning sparse wooden points, where the metric specificity stands for the ratio of true wood points predicted as wood. This value can be elevated to  $79\%$ , but at the cost of decreasing the accuracy metric to  $80\%$ , where the metric accuracy stands for the ratio of correctly predicted wood and leaf points. For better understanding how each class
contributes to the overall performance, we introduced two more informative metrics mentioned in the study of Branco et al. [39], Geometric mean (G-mean) and Balanced Accuracy (BA). These metrics validate the performance of the SOUL model, providing more comprehensive insights into its capabilities.
(a) FSCT
(b) SOUL
(c) Ground truth
Figure 5: Qualitative results on ULS test data. Because of the class imbalance issue, methods such as FSCT[9], LeWos[4], and other existing approaches developed for dense forest point clouds, like TLS or MLS, (cf. Section 2) are ineffective.
4.6 Results on the other data.
After testing on data sets from Sandhausen data set (Weiser et al. [31]) and NSW Forest data set [32], we observed that the SOUL exhibits good generalization capability across various forests. Qualitative results are shown in Figure 6. First, SOUL demonstrates good results on Sandhausen data set [31]. However, SOUL does not outperform the others methods on TLS, but it shows better discrimination within the tree canopy. A possible improvement of SOUL performance is foreseen for denser data provided the model is retrained on similar data sets.
$^{4}$ TP, TN, FP, FN stand respectively True Positives, True Negatives, False Positives, False Negatives.
GVD as a spatial split scheme. Semantic segmentation in tropical forest environments presents distinctive challenges that differ from objects with regular shapes found in data sets such as ShapeNet (Chang et al. [40]) and ModelNet40 (Wu et al. [41]). Forest point clouds are inherently poorly structured and irregular because leaves and branches are small with regard to the LiDAR footprint size, a lot of points derived from LiDAR device are mixed between the two materials of leaves and wood. We propose GVD, an ad-hoc method for ULS data, to overcome the chaotic nature of forest point clouds with a pre-partition step. Similar to the enhancement that RCNN (Girshick et al. [42]) brings to CNN (LeCun et al. [43]) on image classification and semantic segmentation tasks, GVD serves as a region proposal method that provides improved training sample partitioning for deep learning networks. The neural network can benefit from more refined and informative data, leading to enhanced performance in the semantic segmentation task.
(a) Paracou, French Guiana - TLS
(b) NSW, Australia - MLS
(c) Sandhausen, Germany - ULS
Figure 6: Qualitative results on various LiDAR data from different sites.
Rebalanced loss designed for class imbalance issue. In our model, PointNet++ is chosen as the backbone network due to its known robustness and effectiveness in extracting features from point clouds (Qian et al. [44]). But learning from imbalanced data in point cloud segmentation poses a persistent challenge, as discussed by Guo et al. [45]. To address this issue, we employ a rebalanced loss function tailored to this task. Rather than adjusting class weights during training (Lin et al. [13]; Sudre et al. [46]), we adopt a random selection approach to balance the training data during the loss computation. This decision stems from the observation that misprediction of a subset of the minority class can lead to significant fluctuations in the loss value, consequently inducing drastic changes in gradients. In an extreme scenario, wherein a single point represents a positive sample, correct prediction of this point drives the loss towards zero, irrespective of other point predictions. Conversely, misprediction of the single point yields a loss value approaching one.
Table 1: Comparison of different methods
Methods	Accuracy	Recall	Precision	Specificity	G-mean	BA1
FSCT [9]	0.974	0.977	0.997	0.13	0.356	0.554
FSCT + retrain	0.977	1.0	0.977	0.01	0.1	0.505
LeWos [4]	0.947	0.97	0.975	0.069	0.259	0.520
LeWos (SoD2) [22]	0.953	0.977	0.975	0.069	0.260	0.523
SOUL (focal loss [13])	0.942	0.958	0.982	0.395	0.615	0.677
SOUL (rebalanced loss)	0.826	0.884	0.99	0.631	0.744	0.757
1 BA (Balanced Accuracy)  $BA = \frac{1}{2} (\text{Recall} + \text{Specificity})$ .
SoD (Significance of Difference).
SOUL model's generalization capability. Despite being originally developed for tropical forest environments, SOUL demonstrates promising performance when applied to less complex forests. We anticipate that with the increasing prevalence of ULS data, coupled with more high-quality labeled data sets, the performance of SOUL is poised to advance further. Additionally, there is potential for SOUL to be extended to other complicated LiDAR scenes. It is conceivable to envision the development of a universal framework based on SOUL that can effectively handle various types of forest LiDAR data, including ULS, TLS, MLS, and even ALS.
Ablation study of geometric features. An ablation study is conducted to evaluate the impact of individual geometric features proposed in our model. Comparative experiments, as detailed in Figure 7 and Table 2, illustrate the advantages of utilizing multiple precomputed geometric features at various scales. Specifically, we observe that features like linearity and verticality significantly enhance trunk recognition, while PCA1 and Sphericity are effective within the canopy. While PointNet++ possesses the capability to internally learn complex features, it may not precisely replicate our empirically selected geometric attributes. PointNet++ excels at adapting and acquiring sophisticated features from training data, whereas our predefined geometric features offer specific advantages in our model's context.
Table 2: Comparison of single geometric feature and multiple geometric features
Methods	Accuracy	Recall	Precision	Specificity	G-mean	BA1
SOUL (linearity + one scale)	0.962	0.978	0.983	0.31	0.55	0.636
SOUL (unabridged)	0.826	0.884	0.99	0.631	0.744	0.757
(b) Only linearity
Figure 7: Comparing the unabridged model to a downgraded model using a single-scale feature linearity. Figures (c) and (e) represent different perspectives of Figures (a), (b), and (d).
(c) Only linearity
(d) Full version
(e) Full version
Limitations. The current observation is that SOUL may not perform as well as other methods on TLS data, as it was not trained for that. Additionally, SOUL may not perform effectively for trees that significantly deviate from the training data. By incorporating diverse samples during training, the model can mitigate the challenges associated with vegetation disparity.
We present SOUL, a novel approach for semantic segmentation in complex forest environments. It outperforms existing methods in the semantic segmentation of ULS tropical forest point clouds and demonstrates high performance metrics on labeled ULS data and generalization capability across various forest data sets. The proposed GVD method is introduced as a spatial split schema to provide refined training samples through pre-partition. One key aspect of SOUL is the use of the rebalanced loss function, which prevents drastic changes in gradients and improves segmentation accuracy. While SOUL shows good performance for different forest types, it may struggle with significantly different trees without retraining. Future work can focus on improving the performance of SOUL on denser forest point clouds to broaden its applications.
Acknowledgement and Disclosure of Funding
The authors are grateful to Nicolas Barbier and Olivier Martin-Ducup for their contributions in data collection and TLS data preprocessing. We also acknowledge that this work has been partially supported by MIAI@Grenoble Alpes, (ANR-19-P3IA-0003).
[49] C. Liu and M. Belkin, “Accelerating SGD with momentum for over-parameterized learning,” in International Conference on Learning Representations (ICLR), 2020.
[50] D.-A. Clevert, T. Unterthiner, and S. Hochreiter, "Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)," in International Conference on Learning Representations, 2016.
[51] V. Nair and G. E. Hinton, "Rectified Linear Units Improve Restricted Boltzmann Machines," in International Conference on Machine Learning, 2010.
[52] A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein, L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy, B. Steiner, L. Fang, J. Bai, and S. Chintala, “PyTorch: An Imperative Style, High-Performance Deep Learning Library,” in Advances in Neural Information Processing Systems (NeurIPS) 32, pp. 8024–8035, Curran Associates, Inc., 2019.
In this supplementary material, we provide more quantitative results, technical details and additional qualitative test examples. Section B presents more implementation and training details of SOUL, in Section C the efficacy of the GVD algorithm is discussed, while Section D conducts ablation studies to investigate the impact of different loss functions on the model's performance. At last, Section E demonstrates the performance improvement achieved by employing multiple-scale computation of geometric features compared to a single-scale approach. A video showcasing the performance of SOUL is available at this link: here.
As a supplement to Sections 3.4 and 4.4, we provide additional details here.
LiDAR scanner. Table 3 presents a summary of the distinguishing laser characteristics between ULS and TLS. The acquisition of LiDAR data is profoundly impacted by atmospheric characteristics, the LiDAR extinction coefficients exhibit a low impact to atmospheric humidity at both  $905\mathrm{nm}$  and  $1550\mathrm{nm}$  wavelengths (Wojtanowski et al. [47]). ULS uses  $905\mathrm{nm}$  as wavelength, at which the scanning device hast the minimum energy consumption (Brede et al. [3]). Meanwhile, the TLS system operates at a wavelength of  $1550\mathrm{nm}$ , which is more susceptible to the fog impact (Wojtanowski et al. [47]). This poses the issue that the spectral reflectance of leaf and wood is less contrasted (Brede et al. [3]) at the  $905\mathrm{nm}$  wavelength, thereby leading us to employ only point coordinates as input for avoiding the use of intensity information.
Table 3: Laser sensor characteristics
TLS	ULS
RIEGL Scanner	VZ400	miniVux-1UAV
Laser Wavelength (nm)	1550	905
Beam divergence (mrad)	≤ 0.25	≤ 1.6*0.5
Footprint diameter (cm@100m)	3.5	16*5
Pulse duration (ns)	3	6
Range resolution (m)	0.45	0.9
Training details. In addition to the content already illustrated in the main paper, we provide further details regarding the training parameters. First and foremost, sufficient training time is required for the model to achieve desirable performance. Our observation indicates that models trained less than 2,000 epochs are inadequately trained to achieve desirable performance. During the training process, three metrics, AUROC, MCC and Specificity, are considered more informative and insightful. Especially, the metric of specificity is crucial as it measures the ability to accurately discriminate wooden points, which is the primary requirement of our method. Another thing to note is that, contrary to the mentioned practice in the main text of increasing the batch size by a factor of two every 1,000 epochs, we did not strictly follow this restriction during the actual training process. Often, the increase in batch size occurred around 800-900 or 1100-1200 epochs for better using the GPU resources. However, theoretically, this offset should not affect the final performance.
Architecture of DL model. We have made several modifications to the architecture of PointNet++ (MSG) [8]. Firstly, we observed that Adam (Kingma & Ba [48]) outperformed Nesterov SGD (Liu & Belkin [49]) in this task, and ELU (Clevert et al. [50]) activation function was better than ReLU (Nair & Hinton [51]). The fraction of the input units to drop for dropout layer was changed from 0.5 to 0.3, that means the probability of an element to be zeroed is  $30\%$  (Paszke et al. [52]). We also decreased the number of hidden neurons and added two more fully connected layers at the end.
Test with error bar. We have calculated a confidence interval at  $95\%$  confidence level to indicate the uncertainty of our method. In the main body of the paper, it was mentioned that we obtained
data from four flights, and each data collection from these flights allowed us to obtain a labeled ULS data set through label transfer from TLS data. The test data set is composed by 40 trees from the same positions across these four labeled ULS data sets, resulting in a total of 160 trees. We divided further these 160 trees based on their unique treeID to 16 smaller test sets. However, trees with the same treeID are not placed together in the same test set. The calculation of confidence intervals was derived from these 16 test sets. The result is summarized in Table 4 and Table 5. It is clear that SOUL outperforms all other methods on ULS data and achieves state-of-the-art performance on metrics such as Specificity, Balanced Accuracy and G-mean.
Table 4: Comparison of different methods
Methods	Specificity	G-mean	BA1
FSCT [9]	0.13	0.356	0.554
FSCT + retrain	0.01	0.1	0.505
LeWos [4]	0.069	0.259	0.520
LeWos (SoD2) [22]	0.069	0.260	0.523
SOUL (focal loss [13])	0.395	0.615	0.677
SOUL (rebalanced loss)	0.576 ± 0.063	0.651 ± 0.030	0.720 ± 0.027
1 BA (Balanced Accuracy)  $BA = \frac{1}{2} (\text{Recall} + \text{Specificity})$ .
SoD (Significance of Difference).
Table 5: Comparison of different methods
Methods	Accuracy	Recall	Precision
FSCT [9]	0.974	0.977	0.997
FSCT + retrain	0.977	1.0	0.977
LeWos [4]	0.947	0.97	0.975
LeWos (SoD) [22]	0.953	0.977	0.975
SOUL (focal loss [13])	0.942	0.958	0.982
SOUL (rebalanced loss)	0.857 ± 0.014	0.865 ± 0.015	0.988 ± 0.002
Output contains redundant points. To proceed with further data analysis and usage, it is necessary to remove duplicate points present in the output.
C Assessing the efficacy of GVD
The ULS data often covers several hectares, or even hundreds of hectares. The situation in tropical forests is also highly complex, with various types and sizes of trees densely packed together. The significant memory demand makes it nearly impossible to process all the data at once, leading us to adopt a spatial split scheme approach as a necessary compromise.
We can select data randomly from the whole scene, but selecting data randomly can result in a sparse and information-poor sample. An alternative is to employ a divide and conquer strategy to handle the chaotic, big volume, and complex ULS data. That is why we propose GVD, a method that involves breaking down the data into more manageable subsets (see Figure 8), allowing us to handle the intricacies and extract meaningful insights in a more systematic manner. This approach enables us to retain the information-rich aspects of the data while overcoming computational challenges associated with the sheer volume of data.
Prior to employing the GVD method, we initially adopted a more intuitive approach. The data was partitioned in unit of voxel, serving as component for batch preparation through down-sampling. However, this approach gave rise to border effects, particularly impeding SOUL's focus on the meticulous segmentation of intricate branch and leaf within tree canopy (see Figure 9(c)). The segmentation of cubes led to the emergence of noise point clusters along the voxel edges. ULS have more points on tree canopy, so the presence of noise point clusters on voxel edges is more severe on tree canopy, which imposes bigger obstacles to our leaf/wood segmentation task.
(a) Comp N°1148
(b) Comp N°575
(c) Comp  $\mathrm{N}^{\circ}1137$
(d) Comp N°552
(e) Comp N°1148
(f) Comp N°575
(g) Comp N°1137
Figure 8: The figure depicts distinct samples subjected to GVD processing and presents qualitative results within each sample. The upper row illustrates the ground truth for each sample (where brown represents leaves and white represents wood), whereas the lower row exhibits the predictive results generated by SOUL (where blue represents leaves and red represents wood incorporating gradient colors for transitional probability).
(h) Comp N°552
(a) Top view
Figure 9: Downgrade version's output with sliding window. The use of GVD eliminates observed edge effects in this context.
(b) Middle layer
(c) Thin layer
(d) Back view
We experimented with cuboids, a choice aimed at preserving greater semantic information within each component sample and expanding the spatial range for batch selection. Similar to a "sliding window", we can systematically traverse the entire forest with overlapping coverage in this way. But border effects persisted (see Figure 9), prompting the introduction of the GVD method, which led to a substantial improvement. Modifying parameters  $\tau$  and  $\gamma$  adjusts the coverage scope of each component in GVD segmentation. Additionally, tweaking the minimum accepted number of voxels and points, two GVD's configurable thresholds, within each component effectively manages component size. This process aids in the elimination of low-information content outlier point clusters to a certain extent.
A comparison between the result of the downgraded version using sliding window as spatial split schema and the individual-component point performance of full version SOUL is illustrated in Figure 8. Notably, in component  $\mathrm{N}^{\circ}552$ , SOUL effectively discriminates trunk points from leaf points that were previously entirely intertwined, surpassing our initial expectations.
(a) Cross entropy loss
(b) Focal loss
Figure 10: The predictions of SOUL model on raw ULS data training with cross entropy loss function and focal loss function. Blue represents wood points with high probability, while red represents leaf points with high probability. Despite utilizing the focal loss, the model remains overwhelmed by leaf points.
D Ablation study of rebalanced loss
In this section, we mainly discuss the significant benefits introduced by the rebalanced loss. The network is biased towards leaf points when using cross-entropy loss function and the use of focal loss function (Lin et al. [13]) does not yield substantial improvements to the task, as its performance remains unsatisfactory when tested on the raw data set (see Figure 10(a) and Figure 10(b)).
Through a comparison of the specificity curves (see focal loss specificity in Figure 11 versus rebalanced loss specificity in Figure 12) and the training/validation loss curves of focal loss and rebalanced loss (see loss curve of the focal loss in Figure 13 versus loss curve of the rebalanced loss in Figure 14), we observed that focal loss failed to address the issue of class imbalance in our task. Plus, it is important to note that in the loss curve of rebalanced loss, the validation loss curve showed significant fluctuations in Figure 14. As foreseen, this variation occurred because the training process employed the rebalanced loss for model training, while the validation process utilized the cross-entropy loss function. Consequently, the loss value used for backpropagation is derived from the rebalanced loss function, which inherently does not consider the majority of leaf points. So once using cross entropy to calculate the loss, all points within a batch are taken into account, which can lead to fluctuations in the validation loss curve. However, it was expected that the validation loss curve would eventually converge on the validation set to demonstrate that using rebalanced loss for backward propagation effectively balances the representation of leaf and wood points. In fact, the final results surpassed focal loss by a significant margin, as evidenced by the convergence of the validation loss curve for rebalanced loss (see Figure 14). For the loss curve of focal loss in Figure 13, both the training and validation processes employ the focal loss function.
Figure 11: Specificity - focal loss (single-scale feature calculation). In the figure, the cyan curve represents the specificity curve of the training data set, while the gray curve represents the specificity curve of the validation data set.
Figure 12: Specificity - rebalanced loss (single-scale feature calculation), cyan curve - training data set, gray curve - validation data set.