## 4 The M_HERACLES Toolbox
### Step 1: Using GIS to Aid the Segmentation of Large Area Point Clouds into Clusters of Objects
#### 4.1.3 Comparison with a Commercial Solution  
In order to assess the quality of our developed approach, a comparison was performed with the automatic point cloud classification results of the commercial software Agisoft Metashape. While Metashape is chiefly a photogrammetric software known for its use in image-based reconstruction, it was also recently augmented with a function for multi-class point cloud semantic segmentation. According to the official documentation, Metashape employs a machine learning technique to perform this task; indeed, it asks its users to submit training datasets in order to improve the classification quality in the future. For the purposes of our comparison, Metashape version 1.5.3 build 8469 (release date 24 June 2019) was used. The Metashape automatic classification was performed on both main datasets: Kasepuhan and St-Pierre. Visual results for the Kasepuhan can be seen in Figure 6. Three classes were defined, namely the 'buildings', 'walls' and 'trees' classes. In Metashape, this corresponds respectively to 'buildings','man-made objects', and 'high vegetation' classes. Figure 6 shows that the Metashape automatic classification had difficulties in distinguishing between buildings and walls, with most of the walls classified as buildings. Some parts of the walls were also classified as high vegetation. This may be due to the fact that the Kasepuhan dataset presented a large-scale and thus more complex scene not entirely suitable for the machine learning-trained function. Unfortunately there is no way to verify this hypothesis since Metashape understandably does not divulge their machine learning method in detail. On the contrary, M_HERACLES managed to classify the objects fairly well thanks to the use of shapefiles to guide it. Visually, Figure 6c also showed that some of parts were nevertheless unclassified, notably the walls at the back of the dataset. This may be due to the low resolution of the point cloud for this part of the site (note the same observation on Metashape results). \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Object**} & \multicolumn{2}{c}{**Point Number**} & \multicolumn{2}{c}{**Misclassed**} & \multirow{2}{*}{**True Positive**} & \multirow{2}{*}{**\%Unclassed**} & \multirow{2}{*}{**\%P**} & \multirow{2}{*}{**\%R**} & \multirow{2}{*}{**\%F1**} \\ \cline{2-2} \cline{6-9}  & **Manual** & & & & & & & & & \\ \hline COLFOCH & 34 011 & 32 384 & 217 & 1 844 & 32 167 & 5.69 & 99.33 & 94.58 & 96.90 \\ \hline STPIERRE & 36 858 & 34 960 & 757 & 2 655 & 34 203 & 7.59 & 97.83 & 92.80 & 95.25 \\ \hline DRIMPOTS & 52 520 & 56 586 & 6 099 & 2 033 & 50 487 & 3.59 & 89.22 & 96.13 & 92.55 \\ \hline PLSUSTICE & 81 074 & 69 559 & 637 & 12 152 & 68 922 & 17.47 & 99.08 & 85.01 & 91.51 \\ \hline PLSFETES & 37 663 & 35 823 & 0 & 1 840 & 35 823 & 5.14 & 100.00 & 95.11 & 97.50 \\ \hline PLRSHIN & 84 833 & 74 738 & 1 026 & 11 121 & 73 712 & 14.88 & 98.63 & 86.89 & 92.39 \\ \hline  & & & & & & **Mean** & **9.06** & **97.35** & **91.75** & **94.35** \\ \hline  & & & & & & **Median** & **6.64** & **98.86** & **93.69** & **93.90** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: The quantitative analysis on the results of step 1 for St-Pierre. In this table, only one class was taken into account (public buildings) with a total of 6 out of 17 objects used in the statistical analysis. %P is precision, %R is recall, and %F1 is the normalized F1 score. Table 3 displays a quantitative comparison of the two tested algorithms for Kasepuhan, also visualized via histograms in Figure 7. M_HERACLES managed to outperform Metashape in most cases (yielding a slightly lower F1 score in the trees class), especially in the walls class. The median value of the F1 score for M_HERACLES was 85.30% compared to Metashape's 60.52%. It showed a lower recall value and higher precision, which may be explained by the fact that the use of shapefiles disproportionately increased Type I error.