# Machine Learning-Powered Rainfall-Based Landslide Predictions in Hong Kong--An Exploratory Study
## 3 The Pilot Study--ML-Powered Natural Terrain Landslide Susceptibility Analysis
### The Machine Learning-Based Modelling
#### 3.6.3 Model Performance Evaluation  
Evaluation metrics serve as tools to measure the performance of the ML models. Among others, accuracy statistics, e.g., accuracy, sensitivity, specificity, threat score, odds ratio, etc., are widely adopted. These accuracy statistics are determined with reference to the classes of true positive (TP), false positive (FP), true negative (TN), and false negative (FN) in a confusion matrix (see Figure 9). A pre-determined cut-off value, or classification threshold that defines the splitting of the model predictions in binary classification, is required in determining those classes. Whilst a classification threshold of 0.5 is often adopted for other statistical-based landslide susceptibility studies that involve data sampling to achieve a 1:1 ratio of landsling and non-landsliding data points, the determination of the threshold in this pilot study is less trivial given the modelling approach adopted. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Feature Set**} & \multirow{2}{*}{**Rainfall**} & \multicolumn{4}{c}{**Machine Learning Algorithm**} \\ \cline{3-5}  & & **Decision Tree** & **Random Forest** & **XGBoost** \\ \hline \multirow{3}{*}{FS1} & 24 h-NMRR & FS1-24-DT & FS1-24-RF & FS1-24-XGB \\  & 4 h-NMRR & FS1-4-DT & FS1-4-RF & FS1-4-XGB \\  & 4 h- \& 24 h-NMRR & FS1-4k24-DT & FS1-4k24-RF & FS1-4k24-XGB \\ \hline \multirow{3}{*}{FS2} & 24 h-NMRR & FS2-24-DT & FS2-24-RF & FS2-24-XGB \\  & 4 h-NMRR & FS2-4-DT & FS2-4-RF & FS2-4-XGB \\ \cline{1-1}  & 4 h- \& 24 h-NMRR & FS2-4k24-DT & FS2-4k24-RF & FS2-4k24-XGB \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Summary of Analysis Cases Considered in the Pilot Study. In view of this fact, the performances of the ML models in this pilot study were evaluated using the Area Under Curve (AUC) of the receiver operating characteristic (ROC) curves (see Figure 9). An ROC curve plots the true positive rate (TPR) against the false positive rate (FPR) for different classification thresholds, such that the pre-determination of the threshold is not required. A higher ROC AUC value indicates the better performance of the model over the whole range of the classification threshold. The ROC AUC of the ML models as assessed based on the input dataset (referred to as training data here) and the two sets of testing data (TD1 & TD2) are summarized in Table 7. The key observations from the evaluation results are summarized as follows:  
* The ROC AUC based on the training data of all cases are over 94%, indicating that the machine learning models fit the training data very well. * The ROC AUC based on the two sets of testing data are over 86%. The highest ROC AUCs based on TD1 and TD2 were up to 91.5% and 97.3%, respectively, for individual XGBoost models. The ROC AUC based on TD1 is obviously lower than that based on TD2 for all of the cases as the former served as a more stringent test of the models' ability to make forward predictions. All in all, the ROC AUC values that were achieved reveal that all of the ML models are able to make fairly accurate predictions. * With reference to the ROC AUC based on the testing data, the XGBoost and Random Forest models perform better than the Decision Tree models. The introduction of additional features (i.e., FS2) improved the performance of the ML models; the effect is more obvious when the models were tested with TD2. On the other hand, the durations of NMRR seemed to have no significant effect on the models' performances. The second level of the model performance evaluation will focus on analysis model FS2-24-XGB which shows the highest ROC AUC for both TD1 and TD2. (b) Susceptibility Map  
Spatial Resolution  
Figure 10 shows the landslide susceptibility maps near the Tai O area as predicted by ML models FS1-24-XGB and FS2-24-XGB under a hypothetical constant rainfall scenario with a 24-h NMRR of 0.225, corresponding to the average 24-hr NMRR Class IV in Ko & Lo (2016) [2].