# Machine Learning-Powered Rainfall-Based Landslide Predictions in Hong Kong--An Exploratory Study
## 3 The Pilot Study--ML-Powered Natural Terrain Landslide Susceptibility Analysis
### The Machine Learning-Based Modelling
#### 3.6.3 Model Performance Evaluation
The locations of the recent landslides, including those that happened in the years from 1985 to 2008 considered in the model construction, are included to indicate  
Figure 9: Definition of ROC AUC and Confusion Matrix. areas with higher landslide susceptibility. The positive classification scores are directly adopted as the predicted landslide susceptibility. It is observed that most of the recent landslides fall within areas with a higher landslide susceptibility for both models. On the other hand, the FS2-24 hr-XGB model can distinguish landslide susceptibility with a much higher resolution as compared with FS1-24 hr-XGB. While Figure 10 allows for a snapshot appreciation of the landslide susceptibility forecast ability of the two ML models adopting different feature sets visually, Figure 11 provides a more comprehensive picture of the degree of spatial resolution improvement by comparing the range of predicted landslide susceptibility values of the entire pilot study area under different rainfall intensities. The rainfall intensities correspond to the mean normalized 24-h maximum rolling rainfall intensities of 24-h NMRR Classes I to V considered in Ko & Lo (2016) [2]. It is revealed that the introduction of the three additional features (i.e., plan curvature, profile curvature, and upslope catchment area) improves the overall resolution of the landslide susceptibility map by approximately three orders of magnitude. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Evaluation Data** &
\begin{tabular}{c} **Feature Set \&** \\ **NMRR Duration** \\ \end{tabular} & **Decision Tree** & **Random Forest** & **XGBoost** \\ \hline \multirow{4}{*}{Training Data} & FS1-24 & 0.9571 & 0.9881 & 0.9660 \\  & FS1-4 & 0.9426 & 0.9905 & 0.9559 \\  & FS1-4\&24 & 0.9601 & 0.9965 & 0.9748 \\  & FS2-24 & 0.9674 & 0.9965 & 0.9808 \\  & FS2-4 & 0.9480 & 0.9931 & 0.9720 \\  & FS2-4\&24 & 0.9599 & 0.9999 & 0.9769 \\ \hline \multirow{4}{*}{Testing Data 1} & FS1-24 & 0.8770 & 0.8836 & 0.8867 \\  & FS1-4 & 0.8709 & 0.8740 & 0.8873 \\  & FS1-4\&24 & 0.8717 & 0.8742 & 0.8602 \\ (TD1) & FS2-24 & 0.8764 & 0.9083 & 0.9149 \\  & FS2-4 & 0.8744 & 0.8970 & 0.9130 \\  & FS2-4\&24 & 0.8795 & 0.9042 & 0.9132 \\ \hline \multirow{4}{*}{Testing Data 2} & FS1-24 & 0.9499 & 0.9593 & 0.9627 \\  & FS1-4 & 0.9444 & 0.9654 & 0.9695 \\ \cline{1-1}  & FS1-4\&24 & 0.9481 & 0.9563 & 0.9602 \\ (TD2) & FS2-24 & 0.9443 & 0.9670 & 0.9732 \\ \cline{1-1}  & FS2-4 & 0.9437 & 0.9603 & 0.9685 \\ \cline{1-1}  & FS2-4\&24 & 0.9518 & 0.9729 & 0.9734 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: ROC AUC of the Predictive Models. Figure 10: Predicted Landslide Susceptibility of the Tai O Area: (**a**) Model FS1-24-XGB under 24-h-NMRR of 0.225; (**b**) Model FS2-24-XGB under 24-h-NMRR of 0.225. Legend covers the range of landslide susceptibility predicted for the entire pilot study area. No. of Actual Landslides by Susceptibility of Area  
[PERSON] et al. (2021) [14] suggested looking at the accuracy of an ML model by validating the areal extent of each susceptibility class against the landslide density distribution form the landslide inventory. A model is accurate when the landslide density ratio increases moving from low susceptibility classes to high susceptibility classes, and when the high susceptibility classes cover only a small extent of areas. For this purpose, Figure 12 plots the actual number of landslides from years 1985 to 2008 as recorded in the ENTLI, and the distribution of area of the study area in each year, by the positive classification score (i.e., predicted landslide susceptibility) given in FS2-24-XGB. About 85% (3695 out of 4377 nos.) of the historical landslides that occurred between 1985 and 2008 fall within 4% of the area with the highest landslide susceptibility (i.e., positive classification score \(\geq 10^{-4}\)), demonstrating that the ML model is making fairly accurate landslide susceptibility predictions. Figure 11: Ranges of Predicted Landslide Susceptibility.