# Hybrid Early Warning System for Rock-Fall Risks Reduction
## 5 Results and Discussion
### Model Validation  
This section summarizes the findings of system models validation. The proposed system was validated using four performance measures: sensitivity, specificity, accuracy, and reliability. First, the prediction model's overall prediction performance measures based on a confusion matrix (see Table 7) were evaluated for training and validation data sets. The confusion matrix was created for both training and testing. The confusion matrix was used to calculate sensitivity, specificity, and accuracy. In the above table, true positive (TP) means all events were true detected, false negative (FN) means some events occurred but were not detected, true negative (TN) means events were absent and the system reported an absent event, and false positive (FP) means an event was absent but the system reported it as present. The result shows that the average sensitivities of training and validation data were 70.4% and 71.4%, respectively. That means, even for the lowest sensitivity levels, only 29.6% of the rock-fall events were not detected correctly. The average specificities were about 86.3% and 86.5%, respectively, which means the system had a high ability to disregard fake events. The accuracies were 79.9% and 81.0% for the training and the validation data. The reliability was 0.79. Next, the monitoring model performance measures were obtained by testing the system 180 times with a rock with the of size 78 cm\({}^{3}\). The tests were divided into nine periods, and 20 tests were assigned for each period. In each period, sensitivity, specificity, and accuracy were calculated. Table 8 illustrates the results for all test cases. \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Data Type**} & \multirow{2}{*}{
\begin{tabular}{c} **Observed** \\ **Rock-Fall Even** \\ \end{tabular} } & \multicolumn{3}{c}{**Predicted**} \\  & & **Rock-Fall Even** & **Percentage C** \\ \cline{3-5}  & & **Not occur 0** & **Occurs 1** & \\ \hline \multirow{3}{*}{**Training Data**} & Not occur 0 & TN = 69 & FP = 11 & 86.3\% \\  & Occurs 1 & FN = 16 & TP = 38 & 70.4\% \\ \cline{3-5}  & Overall Percentage & & 79.9\% \\ \cline{3-5}  & Not occur 0 & TN = 32 & FP = 5 & 86.5\% \\ \cline{3-5}  & Occurs 1 & FN = 6 & TP = 15 & 71.4\% \\ \cline{3-5}  & Overall Percentage & & 81.0\% \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Confusion matrix. Table 8 illustrates that the average sensitivity of the proposed method was about 88.8%, which means that, even for the lowest levels of sensitivity, only 1.2% of the rock-fall events were not detected correctly. This indicates that the system had a high sensitivity in detecting and tracking rocks. The average specificity of the proposed method was about 92.2%, which means the system had a high ability to distinguish between real and fake events. The average accuracy was 90.6. In this work, reliability was calculated according to accuracy values from Table 8, and, by using Equation (11), we obtained the system reliability equal to 0.9. That means the system had high reliability in detecting and tracking rocks and indicates that the system was valid. Finally, the hybrid model performance measures were obtained based on its sub-models' effects (prediction model and monitoring model). The result shows that the average sensitivity was 96.7%. That means, even for the lowest sensitivity levels, only 3.3% of the rock-fall events were not detected correctly. The proposed method's average specificity was 99.1%, which means the system had a high ability to disregard fake events. The accuracy of 97.9% and a reliability of 0.98 indicate the goodness and the stability of the hybrid model. In another way, the model indicates high consistency. By using the proposed hybrid model, the average risk probability was reduced from 6373 \(\times\) 10\({}^{-4}\) to 1.13 \(\times\) 10\({}^{-8}\). When comparing the hybrid model results to the monitoring and the prediction models, it must be pointed out that the proposed model outperformed the existing models.