# Machine Learning for Classification of AN Eroding Scarp Surface Using Terrestrial Photogrammetry with NIR and RGB Imagery

H. Bernsteiner, N. Brožová, I. Eischeid, A. Hamer, S. Haselberger, M. Huber, A. Kollert, T. M. Vandyk, F. Pirotti

<figure><img src="image_1.png" /></figure>

## To cite this version:

H. Bernsteiner, N. Brožová, I. Eischeid, A. Hamer, S. Haselberger, et al., Machine Learning for Classification of AN Eroding Scarp Surface Using Terrestrial Photogrammetry with NIR and RGB Imagery. ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 2020, 5.3, pp.431-437. 10.5194/isprs-annals-V-3-2020-431-2020. insu-03712899

HAL Id: insu-03712899

https://insu.hal.science/insu-03712899

Submitted on 4 Jul 2022

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
MACHINE LEARNING FOR CLASSIFICATION OF AN ERODING SCARP SURFACE USING
TERRESTRIAL PHOTOGRAMMETRY WITH NIR AND RGB IMAGERY

H. Bernsteiner <fnref n="1" />, N. Brožová <fnref n="2" />, I. Eischeid <fnref n="3" />, A. Hamer <fnref n="4" />, S. Haselberger <fnref n="5" />, M. Huber <fnref n="6" />, A. Kollert <fnref n="7" />, T. M. Vandyk <fnref n="8" />, F. Pirotti <fnref n="9" />

Commission III, III/10

**KEY WORDS:** Terrestrial Photogrammetry, Structure from Motion, Surface Classification, Machine Learning, High Mountain Environment

ABSTRACT:

Increasingly advanced and affordable close-range sensing techniques are employed by an ever-broadening range of users, with varying competence and experience. In this context a method was tested that uses photogrammetry and classification by machine learning to divide a point cloud into different surface type classes. The study site is a peat scarp 20 metres long in the actively eroding river bank of the Rotmoos valley near Obergurgl, Austria. Imagery from near-infra red (NIR) and conventional (RGB) sensors, georeferenced with coordinates of targets surveyed with a total station, was used to create a point cloud using structure from motion and dense image matching. NIR and RGB information were merged into a single point cloud and 18 geometric features were extracted using three different radii (0.02 m, 0.05 m and 0.1 m) totalling 58 variables on which to apply the machine learning classification. Segments representing six classes, dry grass, green grass, peat, rock, snow and target, were extracted from the point cloud and split into a training set and a testing set. A Random Forest machine learning model was trained using machine learning packages in the R-CRAN environment. The overall classification accuracy and Kappa Index were 98% and 97% respectively. Rock, snow and target classes had the highest producer and user accuracies. Dry and green grass had the highest omission (1.9% and 5.6% respectively) and commission errors (3.3% and 3.4% respectively). Analysis of feature importance revealed that the spectral descriptors (NIR, R, G, B) were by far the most important determinants followed by verticality at 0.1 m radius.

1. INTRODUCTION

In the past decades a step change in close range remote sensing
technologies has allowed techniques such as photogrammetry to
be employed by an increasingly diverse range of users, not only
the specialist (Eltner et al., 2016; Westoby et al., 2012). The
inevitable result of this proliferation has been an abundance of
high-quality data for which automated processes of classification
have become a practical necessity (Grilli et al., 2017), since
manual labelling and classification are cost- and time-demanding
and unfeasible for large datasets. In this context, at the 2019
Innsbruck Summer School, Obergurgl (Rutzinger et al., 2018,
2016), a team of researchers applied machine learning (ML) to a
point cloud derived from dense image matching of a terrestrial
photogrammetric survey. This came as part of a larger survey in
in a mountain environment with also a remotely piloted aircraft
system (RPAS) over the whole valley (Scaioni et al., 2018). Near
infrared and RGB information was collected from both RPAS
and terrestrial surveys, as previous literature has largely proven

that any vegetation component can validly be labelled with
spectral features (Alba et al., 2011).

The fields of interest of the participants comprise a diversity of
applications that can benefit from close-range sensing: from
primary colonisation of recently deglaciated ground, through
slope stability and evolution, to the surveying and interpretation
of rarely preserved 700-million-year-old landforms. These users
represent some of the numerous examples that may benefit from
common data manipulation techniques, allowing statistical data
to be derived from classifications within point cloud data.

Within this study the aim was to i) classify relevant surface types
within a small section of a mountain valley floor, ii) compare the
efficacy of optical and geometric properties in distinguishing
between key surface types and finally iii) to evaluate
photogrammetric methods and machine learning approaches with
respect to the group members' research interests.

<footnote n="1">¹ University of Bayreuth, Chair of Geomorphology, 95447 Bayreuth, Germany - Heidemarie.Bernsteiner@uni-bayreuth.de</footnote>

<footnote n="2">² WSL Institute for Snow and Avalanche Research SLF, Flüelastrasse 11, 7260 Davos Dorf, Switzerland - natalie.brozova@slf.ch</footnote>

<footnote n="3">³ Department of Arctic and Marine Biology, UiT The Arctic University of Norway, 9037 Tromsø, Norwegian Polar Institute, Fram Center, Hjalmar Johansens gate 14, 9007 Tromsø, Norway, Aarhus University, Department of Bioscience, Denmark - isabell.eischeid@uit.no</footnote>

<footnote n="4">⁴ University of Manchester, Geography, School of Environment Education & Development, Manchester M13 9PL, UK - alexandra.hamer@manchester.ac.uk</footnote>

<footnote n="5">⁵ University of Vienna, Institute of Geography and Regional Research, 1010 Vienna, Austria. stefan.haselberger@univie.ac.at</footnote>

<footnote n="6">⁶ Centre de Recherches Pétrographiques et Géochimiques, 15 rue Notre Dame des Pauvres, 54500 Vandœuvre les Nancy, France -
marius.huber@univ-lorraine.fr</footnote>

<footnote n="7">⁷ Institute for Interdisciplinary Mountain Research, Austrian Academy of Sciences, Technikerstr. 21a, 6020 Innsbruck, Austria -
andreas.kollert@oeaw.ac.at</footnote>

<footnote n="8">⁸ Department of Geography, Royal Holloway University of London, Egham, Surrey, TW200EX, UK - thomas.vandyk.17@ucl.ac.uk</footnote>

<footnote n="9">⁹ CIRGEO Interdepartmental Research Center in Geomatics, TeSAF Department, University of Padova, 35020 Legnaro (PD), Italy -
francesco.pirotti@unipd.it</footnote>
A terrestrial photogrammetry survey was undertaken on a partially snow-covered river bank comprising peat, loose soil, rock and vegetation and these different components were each assigned a class. Point cloud segments representative of each were used for training a machine learning model, subsequently used to classify areas within the entire point cloud with a degree of reliability.

## 2. STUDY SITE

The study area is located at the main alpine divide of the Austrian Alps at the border between the State of Tyrol (Austria) and the Province of South Tyrol (Italy). The Rotmoos valley (46° 50' 24" N, 11° 01' 59" E) extends c. 6 km from SE to NW and covers an area of c. 1 km² with an altitudinal range from c. 2240 m to c. 3400 m. The area is characterised by an inner alpine climate and surrounded and protected by mountains. The nearby weather station (Obergurgl, 1938 m) shows a low mean annual precipitation of c. 819 mm, with maxima from June to August. Mean annual air temperature is + 2.2 °C, with the highest monthly means of around + 16 °C in July and the lowest mean of -8.3 °C in February (data period 1971-2000; ZAMG - Austria's national weather service 2018).

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /><figcaption>Figure 1. Location of study area (red star) and Obergurgl (circle on middle image). The bottom image shows a close-up of the erosion feature surveyed for this study.</figcaption></figure>

During the last glacial, the valley was shaped by glacial erosion through multiple advances of the Rotmoos glacier. The last glaciation of the valley floor was during the Younger Dryas period. After the retreat of the Younger Dryas glacier, the valley was filled with up to 40 m of sediment (Patzelt, 1995) and remained ice free during the last re-advance of the Rotmoos glacier during the Little Ice age. The attributed prominent terminal moraine complex is located c. 1 km up valley from the study site. In the distal and central part of the valley, a peat bog developed that covers an area of c. 800 by 120 m. The peat deposits are up to 2.65 m thick and radiocarbon dates from the base and top of the peat are c. 5994 and 1629 years before present respectively (Bortenschlager, 2010). Today, the peat bog is dissected and eroded by the river Rotmoosache, a tributary of the river Ötztaler Ache. A c. 20 m stretch of its bank is the object of this study (for location see Figure 1, red star). The study section comprises steep peat faces, which are highly water saturated and partly covered by snow and vegetation (Figure 1 bottom).

## 3. METHODS

Data acquisition was planned together with a team that acquired UAV imagery. Three of their ground control points (GCPs) were measured with differential GNSS (Global Navigation Satellite System) in order to georeference the final product in a projected coordinate system. A control measurement between points revealed sub-centimetre accuracy of the GCPs. In the study area for this investigation eleven GCPs were placed on the eroding scarp (Figure 1 bottom) and georeferenced using a total station positioned on one of the measured UAV GCPs.

Terrestrial photogrammetry was used to survey the eroding scarp surface. Images were acquired using a consumer-grade RGB camera Canon EOS 450D (27mm) and a NIKON D-200 with HOYA R72 filter, modified to operate in the Near-Infrared region (NIR) of the electromagnetic spectrum (750 – 1,500 nm). The modification allowed the CCD sensor to record reflected radiation above 720 nm. As shown in Figure 2, the natural sensitivity of the CCD sensor includes wavelengths up to 950 nm, but these are filtered out by the camera filter. By removing this filter and adopting an external filter, NIR information can be recorded in the image.

<figure><img src="image_5.png" /><figcaption>Figure 2. CCD sensitivity of Nikon D200.</figcaption></figure>

### 3.1 Pre-processing

In total 24 images were imported to Agisoft Metashape (AM). The GCPs were automatically detected and located by the
software. The coordinates of the GCPs that were measured using the total station in the field, were loaded into AM. After camera alignment, dense image matching was performed in order to obtain a dense point cloud. The same workflow was applied to the NIR imagery acquired with the modified camera.

The point clouds from the RGB and NIR imagery were imported to CloudCompare software. First, NIR information was merged to the RGB point cloud using three nearest neighbours. The software finds, for each point in the RGB cloud, the three closest points in the NIR cloud, and appends the average from the three NIR values to the RGB point. Additionally, 18 geometric features were calculated within CloudCompare (see Table 1). A description on the computation of the eigenvalue and vector based features is given by Hackel et al. (2016).

<table><caption>Table 1. List of geometric features calculated from the point cloud.</caption><tr><td>1. Roughness</td><td>2. Mean curvature</td></tr><tr><td>3. Gaussian curvature</td><td>4. Normal change rate</td></tr><tr><td>5. Number of neighbours</td><td>6. Surface density</td></tr><tr><td>7. Volume density</td><td>8. Eigenvalues sum</td></tr><tr><td>9. Omnivariance</td><td>10. Eigenentropy</td></tr><tr><td>11. Anisotropy</td><td>12. Planarity</td></tr><tr><td>13. Linearity</td><td>14. PCA1</td></tr><tr><td>15. PCA2</td><td>16. Surface variation</td></tr><tr><td>17. Sphericity</td><td>18. Verticality</td></tr></table>

The geometric features are calculated by considering a number of neighbours. In CloudCompare the neighbours are identified using a user-defined radius. In this work three radii were tested: 0.02 m, 0.05 m and 0.1 m. Respectively each distance had the following number of neighbouring points – average (standard deviation): 4.8(3.1), 12(8), 24.12(16). The final cloud had ~2.16 million points. The final cloud was exported as a text file with information on the coordinates (x, y, z), RGB and NIR values and the 18 geometric features for each radius. The final feature count was therefore 54 geometric features and four spectral features (NIR, R, G, B), for a total of 58 descriptive features that can be used for classification. Further analysis was carried out using the statistical software R and R Studio.

## 3.2 Classification

The final cloud was imported in a text file format as a table (data.frame) in R 3.6 (R Core Team, 2018). A random forest classifier was used as previous tests have shown positive results (Pirotti et al., 2016; Pirotti and Tonion, 2019) and initial tests run on the data of this study supported the use of this classifier. This choice is however debatable as many factors must be taken into consideration and the issue is expanded upon in the Discussion Section. For the random forest model, the number of trees in the ensemble was set to 200 and the number of variables to split a node was set to 16, after tuning the model trying a grid of 6x6 reasonable values of number of trees and number of variables. Six surface classes were defined, *dry grass*, *green grass*, *peat*, *rock*, *snow* and *target* (Table 2). This last class is represented by the 11 black and white targets used for GCPs.

For classification with machine learning (ML), manually labelled (classified) points were used for training and testing. Manual labelling was a crucial task. For this study, subsets for each of the six classes were extracted from the original point cloud by manually clipping regions with points having a defined unique class. Table 2 shows that the number of labelled points per class is quite balanced, except for the “rock” class. The rock class is under-represented in the study area, as the surveyed area is

mostly covered with grass, peat or snow. It was nevertheless included as it does represent a class of its own and cannot be reasonably merged in the other classes.

<table><caption>Table 2. List of classes with number of points in the labelled subset and colour related to Figure 4.</caption><thead><tr><th>ID</th><th>Class</th><th>N. Points</th><th></th></tr></thead><tbody><tr><td>1</td><td>dry grass</td><td>10932</td><td></td></tr><tr><td>2</td><td>green grass</td><td>6494</td><td></td></tr><tr><td>3</td><td>peat</td><td>13344</td><td></td></tr><tr><td>4</td><td>rock</td><td>1621</td><td></td></tr><tr><td>5</td><td>snow</td><td>5476</td><td></td></tr><tr><td>6</td><td>target</td><td>12198</td><td></td></tr></tbody></table>

The labelled point set was further split into training (50%) and testing (50%) subsets. Random Forest was used to create a classification model based on the training data. The efficacy of each variable in creating the model was reported as Mean Decrease Accuracy (MDA). To assess accuracy of the model, the points in the test dataset were classified and the predicted classes compared to the labelled classes using a confusion matrix and accuracy metrics. Finally, the fitted Random Forest model was used to classify the entire point cloud, to generate a labelled 3D model of the study area (Figure 4).

## 4. RESULTS

The machine learning approach provided a very high overall classification accuracy of 98% across all classes, with a Kappa index of 97%. These figures are related to the independent testing dataset. Predictor accuracy was highest for the following classes: target, peat and snow, followed by rock and dry vegetation. Green and dry vegetation had both highest commission and omission errors, thus showing a likely mutual misclassification. Accuracy was high for snow, rock, peat and targets (> 98%). However, when dividing vegetation into dry and green, the observer’s accuracy drops to below 98.1% and 94.4% respectively (Table 3).

<table><caption>Table 3. Confusion matrix: columns=IDs of predicted classes, rows=IDs of real classes, commission errors (red) and omission errors (green) for each class. Class names are listed in Table 2.</caption><thead><tr><th></th><th>1</th><th>2</th><th>3</th><th>4</th><th>5</th><th>6</th><th>Σ</th></tr></thead><tbody><tr><th>1</th><td>4183</td><td>72</td><td>4</td><td>1</td><td>0</td><td>2</td><td>4262 1.9%</td></tr><tr><th>2</th><td>132</td><td>2384</td><td>9</td><td>0</td><td>0</td><td>0</td><td>2525 5.6%</td></tr><tr><th>3</th><td>5</td><td>11</td><td>5119</td><td>0</td><td>0</td><td>4</td><td>5139 0.4%</td></tr><tr><th>4</th><td>0</td><td>0</td><td>0</td><td>633</td><td>0</td><td>0</td><td>633 0.0%</td></tr><tr><th>5</th><td>0</td><td>0</td><td>0</td><td>0</td><td>2139</td><td>2</td><td>2141 0.1%</td></tr><tr><th>6</th><td>6</td><td>1</td><td>6</td><td>0</td><td>9</td><td>4806</td><td>4828 0.5%</td></tr><tr><th>Σ</th><td>4326</td><td>2468</td><td>5138</td><td>634</td><td>2148</td><td>4814</td><td>19528</td></tr><tr><th></th><td>3.3%</td><td>3.4%</td><td>0.4%</td><td>0.2%</td><td>0.4%</td><td>0.1%</td><td></td></tr></tbody></table>

Feature importance over the whole classification process was also analysed (Figure 3). Figure 3 revealed that spectral descriptors were the most influential in classification. Within these, NIR ranked highest followed by red and then overall RGB. The next most-important non-spectral descriptor was the verticality at 0.1 m radius.
<figure><img src="image_6.png" /><figcaption>Figure 3. Overall variable importance of first ten most important features.</figcaption></figure>

Looking at importance of each feature for class scale (Table 4) shows that verticality at the highest radius size (0.1 m) was particularly important for the “target” and “green grass” classes, along with point density features (features 5, 6 and 7 - i.e. number of neighbours, surface and volume density – see Table 1). Surface variation (16) and sphericity (17) at 0.1 m showed a moderate, but relatively constant importance over all classes.

<figure><img src="image_7.png" /><figcaption>Figure 4. Classified results over the full point cloud (left) and RGB point cloud (right).</figcaption></figure>

<table><caption>Table 4. Variable importance for each class and feature. Numbers in first column are related to Table 1, values in parenthesis in first column is radius for neighbourhood definition.</caption><thead><tr><th>Feature</th><th>dry grass</th><th>green grass</th><th>peat</th><th>rock</th><th>snow</th><th>target</th></tr></thead><tbody><tr><td>R</td><td>1,00</td><td>0,85</td><td>1,00</td><td>0,87</td><td>0,97</td><td>1,00</td></tr><tr><td>G</td><td>1,00</td><td>0,83</td><td>1,00</td><td>0,89</td><td>0,97</td><td>1,00</td></tr><tr><td>B</td><td>0,98</td><td>0,87</td><td>1,00</td><td>0,95</td><td>0,87</td><td>0,98</td></tr><tr><td>NIR</td><td>1,00</td><td>0,92</td><td>0,91</td><td>0,81</td><td>1,00</td><td>1,00</td></tr><tr><td>1 (0.05)</td><td>0,59</td><td>0,59</td><td>0,67</td><td>0,59</td><td>0,59</td><td>0,57</td></tr><tr><td>2 (0.05)</td><td>0,60</td><td>0,60</td><td>0,64</td><td>0,60</td><td>0,60</td><td>0,58</td></tr><tr><td>3 (0.05)</td><td>0,65</td><td>0,65</td><td>0,76</td><td>0,65</td><td>0,65</td><td>0,59</td></tr><tr><td>4 (0.05)</td><td>0,69</td><td>0,69</td><td>0,82</td><td>0,69</td><td>0,69</td><td>0,62</td></tr><tr><td>5 (0.05)</td><td>0,60</td><td>0,83</td><td>0,82</td><td>0,60</td><td>0,60</td><td>0,83</td></tr><tr><td>6 (0.05)</td><td>0,60</td><td>0,83</td><td>0,82</td><td>0,60</td><td>0,60</td><td>0,83</td></tr><tr><td>7 (0.05)</td><td>0,60</td><td>0,83</td><td>0,82</td><td>0,60</td><td>0,60</td><td>0,83</td></tr><tr><td>8 (0.05)</td><td>0,51</td><td>0,54</td><td>0,52</td><td>0,51</td><td>0,51</td><td>0,54</td></tr><tr><td>9 (0.05)</td><td>0,58</td><td>0,58</td><td>0,81</td><td>0,64</td><td>0,58</td><td>0,54</td></tr><tr><td>10 (0.05)</td><td>0,51</td><td>0,54</td><td>0,51</td><td>0,52</td><td>0,52</td><td>0,54</td></tr><tr><td>11 (0.05)</td><td>0,70</td><td>0,70</td><td>0,83</td><td>0,70</td><td>0,70</td><td>0,62</td></tr><tr><td>12 (0.05)</td><td>0,59</td><td>0,59</td><td>0,59</td><td>0,59</td><td>0,59</td><td>0,59</td></tr><tr><td>13 (0.05)</td><td>0,54</td><td>0,59</td><td>0,54</td><td>0,56</td><td>0,54</td><td>0,59</td></tr><tr><td>14 (0.05)</td><td>0,54</td><td>0,57</td><td>0,60</td><td>0,54</td><td>0,54</td><td>0,57</td></tr><tr><td>15 (0.05)</td><td>0,58</td><td>0,59</td><td>0,58</td><td>0,58</td><td>0,58</td><td>0,59</td></tr><tr><td>16 (0.05)</td><td>0,69</td><td>0,69</td><td>0,82</td><td>0,69</td><td>0,69</td><td>0,62</td></tr><tr><td>17 (0.05)</td><td>0,70</td><td>0,70</td><td>0,83</td><td>0,70</td><td>0,70</td><td>0,62</td></tr><tr><td>18 (0.05)</td><td>0,82</td><td>0,94</td><td>0,77</td><td>0,70</td><td>0,71</td><td>0,94</td></tr><tr><td>1 (0.02)</td><td>0,53</td><td>0,53</td><td>0,64</td><td>0,56</td><td>0,53</td><td>0,53</td></tr><tr><td>2 (0.02)</td><td>0,58</td><td>0,58</td><td>0,72</td><td>0,58</td><td>0,58</td><td>0,55</td></tr><tr><td>3 (0.02)</td><td>0,59</td><td>0,59</td><td>0,70</td><td>0,59</td><td>0,59</td><td>0,56</td></tr><tr><td>4 (0.02)</td><td>0,58</td><td>0,58</td><td>0,81</td><td>0,63</td><td>0,58</td><td>0,55</td></tr><tr><td>5 (0.02)</td><td>0,55</td><td>0,82</td><td>0,74</td><td>0,55</td><td>0,55</td><td>0,82</td></tr><tr><td>6 (0.02)</td><td>0,55</td><td>0,82</td><td>0,74</td><td>0,55</td><td>0,55</td><td>0,82</td></tr><tr><td>7 (0.02)</td><td>0,55</td><td>0,82</td><td>0,74</td><td>0,55</td><td>0,55</td><td>0,82</td></tr><tr><td>8 (0.02)</td><td>0,51</td><td>0,54</td><td>0,52</td><td>0,51</td><td>0,51</td><td>0,54</td></tr><tr><td>9 (0.02)</td><td>0,58</td><td>0,58</td><td>0,81</td><td>0,64</td><td>0,58</td><td>0,54</td></tr><tr><td>10 (0.02)</td><td>0,51</td><td>0,54</td><td>0,51</td><td>0,52</td><td>0,52</td><td>0,54</td></tr><tr><td>11 (0.02)</td><td>0,58</td><td>0,58</td><td>0,82</td><td>0,63</td><td>0,58</td><td>0,54</td></tr><tr><td>12 (0.02)</td><td>0,62</td><td>0,66</td><td>0,52</td><td>0,52</td><td>0,61</td><td>0,66</td></tr><tr><td>13 (0.02)</td><td>0,61</td><td>0,65</td><td>0,56</td><td>0,51</td><td>0,62</td><td>0,65</td></tr><tr><td>14 (0.02)</td><td>0,60</td><td>0,60</td><td>0,60</td><td>0,53</td><td>0,62</td><td>0,60</td></tr><tr><td>15 (0.02)</td><td>0,62</td><td>0,66</td><td>0,53</td><td>0,51</td><td>0,61</td><td>0,66</td></tr><tr><td>16 (0.02)</td><td>0,58</td><td>0,58</td><td>0,81</td><td>0,63</td><td>0,58</td><td>0,55</td></tr><tr><td>17 (0.02)</td><td>0,58</td><td>0,58</td><td>0,82</td><td>0,63</td><td>0,58</td><td>0,54</td></tr><tr><td>18 (0.02)</td><td>0,76</td><td>0,87</td><td>0,75</td><td>0,66</td><td>0,65</td><td>0,87</td></tr><tr><td>1 (0.1)</td><td>0,62</td><td>0,62</td><td>0,62</td><td>0,62</td><td>0,62</td><td>0,60</td></tr><tr><td>2 (0.1)</td><td>0,63</td><td>0,61</td><td>0,61</td><td>0,61</td><td>0,61</td><td>0,63</td></tr><tr><td>3 (0.1)</td><td>0,69</td><td>0,71</td><td>0,69</td><td>0,69</td><td>0,69</td><td>0,71</td></tr><tr><td>4 (0.1)</td><td>0,77</td><td>0,77</td><td>0,80</td><td>0,77</td><td>0,77</td><td>0,70</td></tr><tr><td>5 (0.1)</td><td>0,67</td><td>0,92</td><td>0,89</td><td>0,67</td><td>0,67</td><td>0,92</td></tr><tr><td>6 (0.1)</td><td>0,67</td><td>0,92</td><td>0,89</td><td>0,67</td><td>0,67</td><td>0,92</td></tr><tr><td>7 (0.1)</td><td>0,67</td><td>0,92</td><td>0,89</td><td>0,67</td><td>0,67</td><td>0,92</td></tr><tr><td>8 (0.1)</td><td>0,54</td><td>0,57</td><td>0,54</td><td>0,60</td><td>0,56</td><td>0,57</td></tr><tr><td>9 (0.1)</td><td>0,76</td><td>0,76</td><td>0,80</td><td>0,76</td><td>0,76</td><td>0,71</td></tr><tr><td>10 (0.1)</td><td>0,57</td><td>0,60</td><td>0,55</td><td>0,59</td><td>0,56</td><td>0,60</td></tr><tr><td>11 (0.1)</td><td>0,77</td><td>0,77</td><td>0,80</td><td>0,77</td><td>0,77</td><td>0,71</td></tr><tr><td>12 (0.1)</td><td>0,57</td><td>0,57</td><td>0,57</td><td>0,60</td><td>0,57</td><td>0,57</td></tr><tr><td>13 (0.1)</td><td>0,53</td><td>0,54</td><td>0,52</td><td>0,62</td><td>0,53</td><td>0,54</td></tr><tr><td>14 (0.1)</td><td>0,63</td><td>0,60</td><td>0,60</td><td>0,64</td><td>0,60</td><td>0,63</td></tr><tr><td>15 (0.1)</td><td>0,56</td><td>0,57</td><td>0,56</td><td>0,60</td><td>0,56</td><td>0,57</td></tr><tr><td>16 (0.1)</td><td>0,77</td><td>0,77</td><td>0,80</td><td>0,77</td><td>0,77</td><td>0,70</td></tr><tr><td>17 (0.1)</td><td>0,77</td><td>0,77</td><td>0,80</td><td>0,77</td><td>0,77</td><td>0,71</td></tr><tr><td>18 (0.1)</td><td>0,85</td><td>0,98</td><td>0,80</td><td>0,73</td><td>0,79</td><td>0,98</td></tr></tbody></table>
# 5. DISCUSSION

The primary objective of this investigation was to test the performance of a well-known machine learning algorithm, Random Forest, for classification of point clouds from a terrestrial photogrammetric survey. The following discussion will focus on the strengths and weaknesses of the techniques applied and their resulting outputs, allowing suggestions for future improvements. Moreover, consideration will be given to the replicability of the method used herein, to address the respective research questions of other fields of research.

## 5.1 Effectiveness of methods and outputs

Visual comparison of the classified point cloud and original images immediately reveals a striking qualitative similarity, (Figure 4) which is supported by the confusion matrix and the Kappa index of agreement and other accuracy metrics which have high values. It must be noted that the accuracy metrics are calculated over an independent set, but still over a small number of points, i.e. ~50 thousand labelled points from a total of a point cloud with ~2 million points (~2.5%). The points have been chosen from across the dataset, to avoid spatial autocorrelation (see Figure 5), and further splitting into training and testing datasets have been done with stratified random sampling (strata according to classes), thus keeping independency, but still training and testing data are limited to a small dataset. This implies that classification accuracy metrics can be very high, but not necessarily reflect the performance over all the area. A visual analysis from the classified set (Figure 4) shows that some points of snow patches are erroneously classified as targets. This is probably due to similarity in colour (white target and white snow) and in shape of the object, as a snow patch around a 10 cm radius will appear close to flat, just like a target. Since colour and verticality are the most important features (Figure 3), a similarity in these features will result in class mixing.

<figure><img src="image_8.png" /><figcaption>Figure 5. Red points represent labelled points used for training on the whole dataset.</figcaption></figure>

Moreover, there is a lack of point cloud data within the snow patches (Figure 4); a more quantitative approach reveals that, whereas discrimination between rock, snow, soil and vegetation was reliable, the distinction between wet and dry vegetation was more problematic (Table 3).

The weakness discriminating between the two classes of vegetation (green and dry vegetation) is likely due to the gradual boundaries between the two classes, where one blends into the other. This is exacerbated by their physical proximity, as they do not occur in discrete areas of dry and wet vegetation. In any case this distinction between wet and dry is somewhat arbitrary, using qualitative colour choices within the image for the selection of training data. Future workers should consider the ground truthing of wet and dry areas by touch or using a more quantitative approach with moisture detection equipment.

The lack of data cloud points within the snow areas results from the high snow albedo in contrast to the relatively dark remainder of the images, resulting in over-exposure of the former.

This could be overcome by multiple images from the same position using different exposure settings (i.e. ISO, shutter speed, F-stop), or by using a camera with greater bit depth. The former is potentially labour intensive if images require merging by hand before construction of the point cloud, whereas the latter solution is limited by the available camera. Other options might be taking images in the RAW format and using post-processing in order to fix the over-exposed spots or using a polarizing filter, which increases contrasts and the overall colour saturation. These solutions may however be equally labour or cost intensive and have their limitations.

## 5.2 Descriptors

The importance of NIR and red descriptors suggests that the Normalized Difference Vegetation Index (NDVI) could be used as a proxy for changes along the eroding riverbank. Although generally much less important than spectral features, verticality was the highest non-spectral descriptor and especially useful in dividing rocks from other classes.

### 5.2.1 Geometric descriptors
as seen in Table 4, verticality and density-based descriptors like surface and volume density and number of neighbours do have importance for the Random Forest method. It must be noted that correlation does not necessarily indicate causation and therefore importance might not be related to class-intrinsic information, but to a coincidental relationship. For example most targets were placed vertically thus verticality might support classification, yet this would not be useful in a scenario where targets were placed at different angles. An important rule is that machine learning and artificial intelligence in general work as well as the similarity to trained data. Care must be taken when applying a model trained with a dataset which has different characteristics than the dataset to be classified.

### 5.2.2 Neighbourhood size
The questions of ideal radius size and ideal number of neighbours are important when considering descriptors that use neighbours to describe shape and morphology (Pirotti and Tonion, 2019; Weinmann et al., 2015). The ideal method would involve finding the best number of neighbours for each point from a range by adopting a minimal entropy approach. Although effective this requires very intensive calculation as entropy has to be determined for a range of neighbours for each point. For this study therefore three radii were tested, instead of adopting the minimal entropy method. As a compromise, considering that a number of geometric features, including verticality, showed highest importance at the largest chosen radius, (0.1) the study could be extended to test if even larger radii could give better results.

## 5.3 Class definition

In general, the choice of classes within any point cloud will be guided by the research question at hand. However practical limitations of the data may restrict what can be distinguished by the machine learning process. With this in mind it is suggested that future work could attempt unsupervised classification of the point cloud data. On the one hand this may provide insights into the type of further classifications that could reliably be made and on the other it may reveal patterns that are not otherwise obvious. This would overcome a limitation of the technique employed herein. Specifically, spectral attributes (i.e. examination of the photographs) were used to select classes and training segments therefore it is unsurprising that spectral descriptors are the most influential.
## 5.4 Data acquisition

In terms of practicalities two potential obstacles were encountered, both relating to time. First the terrestrial photography of the survey team took place within the UAV flying area of another team, allowing our data to be placed into a broader context. However, both teams worked on the area at the same time, resulting in significant waiting time during which the area could not be accessed whilst imaged from the air. Improved coordination would significantly reduce time in the field improving efficiency. Second, although only a small area was imaged for terrestrial photogrammetry, it was time consuming in terms of processing time. Caution must be applied in similar surveys in determining the ideal scale for surveying the area of interest as a function of required detail and objects to be classified.

## 5.5 Future work

Several aspects can be further investigated. An interesting aspect is the impact of point density on results. As mentioned in the previous section, the processing time does impact significantly on the method. Depending on the study area size and types of classes it is very likely that there is an ideal density below which the classification accuracy drops unacceptably. Addressing this is relatively simple, as the method can be applied to gradually decimated point clouds. The presented workflow could also be preceded by a point cloud segmentation to enable an object-based classification. As outlined by Vosselman, (2013), this would allow for the computation of additional features such as shape, which could be well suited for discriminating targets from other classes. If rocks have a common morphological appearance in the study area due to their transport history, descriptors of size and shape could also improve the classification.

As outlined in Section, 5.2.1, the importance of some features is likely related to peculiarities of the research area. Therefore, future work should also assess the transferability of the proposed method. This could for example include alteration of class definitions, size of the research area and environmental characteristics. Additionally, the study design should be tested for robustness under different weather conditions. At the day of the study most surfaces were relatively dry. Moist weather could change the surfaces reflectance, making it harder to distinguish certain classes (for example dry and green vegetation or rocks and peat).

## 5.6 Relevance to the research interests

The potential applications of these methods are diverse, in terms of both scale and classification type. At the metre to ten-metre scale, classification techniques can distinguish categories of surface features upon boulders, to help elucidate their transport history. At the kilometre scale this method may be used to classify bedrock surfaces using UAV data, that may be host to subglacially striated areas. The advantage of machine learning in this approach is that these surfaces have distinct geomorphological characteristics, of roughness and curvature, but are often in remote areas. The resulting classification could allow targeting of the most likely areas for ground examination. Machine learning techniques based on the Random Forest algorithm are used to detect landslides and to assess landslide susceptibility maps for large regions e.g.Catani et al., 2013; Kim et al., 2018; Stumpf and Kerle, 2011; Taalab et al., 2018). In terms of monitoring and predicting landslide movements, these methods can be of great help for civil protection and risk mitigation.

Distinguishing between biotic and abiotic classes to better understand their relative influences on a recently deglaciated landscape could also be a target for a classification framework of point cloud data. Moreover, integration of the classification with spatial data can be used to investigate the relationship between primary succession and relief. More environmental applications can be distinguishing between healthy and damaged vegetation, for example due to icing events in Arctic tundra environments.

## 6. CONCLUSIONS

During the course of the Summer School the participants captured, constructed and merged geo-referenced point clouds that contained NIR and RGB data respectively and used these to train a machine learning process that segregated the merged cloud into six classes. The target, snow, rock and peat classes were reliable whereas distinguishing between wet and dry vegetation classes was more problematic, likely due to ambiguous training segments. Optical descriptors were far the most important attributes in classification, although, this is pre-determined by the selection of training areas based on visible light properties. Shape features from 3D points clouds bring some improvement over the overall classification results, and this can be further addressed including the laser scanning data (Pirotti, 2019) from the RPAS survey of the area. During data acquisition, processing and analyses each of the participants learned new skills and identified practical applications of those skills in their own study areas.

## ACKNOWLEDGEMENTS

The study at hand was carried out during the Innsbruck Summer School of Alpine Research 2019 (Rutzinger et al., 2018, 2016). The coordinates of the ground control points for the UAV as well as the terrestrial targets were processed and provided by Prof. Marco Scaioni. The use of open source project R-CRAN and the software CloudCompare is acknowledged.

## REFERENCES

Alba, M., Barazzetti, L., Fabio, F., Scaioni, M., 2011. Filtering vegetation from terrestrial point clouds with low-cost near infrared cameras. Ital. J. Remote Sens. 55-75. https://doi.org/10.5721/ItJRS20114325

Bortenschlager, S., 2010. Age determination of sediment core HASENM, Hasenmoos, Austria. https://doi.org/10.1594/PANGAEA.740396

Catani, F., Lagomarsino, D., Segoni, S., Tofani, V., 2013. Landslide susceptibility estimation by random forests technique: Sensitivity and scaling issues. Nat. Hazards Earth Syst. Sci. https://doi.org/10.5194/nhess-13-2815-2013

Eltner, A., Kaiser, A., Castillo, C., Rock, G., Neugirg, F., Abellán, A., 2016. Image-based surface reconstruction in geomorphometry-merits, limits and developments. Earth Surf. Dyn. https://doi.org/10.5194/esurf-4-359-2016

Grilli, E., Menna, F., Remondino, F., Scanning, L., Scanner, L., 2017. A review of point clouds segmentation and classification algorithms. Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci.
XLII, 1-3. https://doi.org/10.5194/isprs-archives-XLII-2-W3-339-2017

Hackel, T., Wegner, J.D., Schindler, K., 2016. Fast Semantic Segmentation of 3D Point Clouds with Strongly Varying Density, in: ISPRS Annals of the Photogrammetry, Remote Sensing and Spatial Information Sciences. https://doi.org/10.5194/isprs-annals-III-3-177-2016

Kim, J.C., Lee, Sunmin, Jung, H.S., Lee, Saro, 2018. Landslide susceptibility mapping using random forest and boosted tree models in Pyeong-Chang, Korea. Geocarto Int. https://doi.org/10.1080/10106049.2017.1323964

Patzelt, G., 1995. The clay pit of Baumkirchen, in: Quaternary Field Trips in Central Europe INQUA. p. 394.

Pirotti, F., 2019. Open software and standards in the realm of laser scanning technology. Open Geospatial Data, Softw. Stand. 4, 14. https://doi.org/10.1186/s40965-019-0073-z

Pirotti, F., Sunar, F., Piragnolo, M., 2016. Benchmark of Machine Learning Methods for Classification of a Sentinel-2 Image. ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLI-B7, 335-340. https://doi.org/10.5194/isprs-archives-XLI-B7-335-2016

Pirotti, F., Tonion, F., 2019. Classification of Aerial Laser Scanning Point Clouds Using Machine Learning: A Comparison Between Random Forest and Tensorflow. ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-2/W13, 1105-1111. https://doi.org/10.5194/isprs-archives-XLII-2-W13-1105-2019

Rutzinger, M., Bremer, M., Höfle, B., Hämmerle, M., Lindenbergh, R., Oude Elberink, S., Pirotti, F., Scaioni, M., Wujanz, D., Zieher, T., 2018. Training in Innovative Technologies for Close-Range Sensing in Alpine Terrain. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. IV-2, 239-246. https://doi.org/10.5194/isprs-annals-IV-2-239-2018

Rutzinger, M., Höfle, B., Lindenbergh, R., Oude Elberink, S., Pirotti, F., Sailer, R., Scaioni, M., Stötter, J., Wujanz, D., 2016. Close-Range Sensing Techniques in Alpine Terrain. ISPRS Ann. Photogramm. Remote Sens. Spat. Inf. Sci. III-6, 15-22. https://doi.org/10.5194/isprs-annals-III-6-15-2016

Scaioni, M., Crippa, J., Corti, M., Barazzetti, L., Fugazza, D., Azzoni, R., Cernuschi, M., Diolaiuti, G.A., 2018. Technical aspects related to the application of SfM photogrammetry in high mountain. ISPRS - Int. Arch. Photogramm. Remote Sens. Spat. Inf. Sci. XLII-2, 1029-1036. https://doi.org/10.5194/isprs-archives-XLII-2-1029-2018

Stumpf, A., Kerle, N., 2011. Combining Random Forests and object-oriented analysis for landslide mapping from very high resolution imagery, in: Procedia Environmental Sciences. https://doi.org/10.1016/j.proenv.2011.02.022

Taalab, K., Cheng, T., Zhang, Y., 2018. Mapping landslide susceptibility and types using Random Forest. Big Earth Data. https://doi.org/10.1080/20964471.2018.1472392

Vosselman, G., 2013. Point cloud segmentation for urban scene classification. ISPRS - Int. Arch. Photogramm. Remote Sens.

Spat. Inf. Sci. XL-7/W2, 257-262. https://doi.org/10.5194/isprsarchives-XL-7-W2-257-2013

Weinmann, M., Jutzi, B., Hinz, S., Mallet, C., 2015. Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers. ISPRS J. Photogramm. Remote Sens. 105, 286-304. https://doi.org/10.1016/j.isprsjprs.2015.01.016

Westoby, M.J., Brasington, J., Glasser, N.F., Hambrey, M.J., Reynolds, J.M., 2012. 'Structure-from-Motion' photogrammetry: A low-cost, effective tool for geoscience applications. Geomorphology 179, 300-314. https://doi.org/10.1016/j geomorph.2012.08.021