# Understanding Natural Disaster Scenes from Mobile Images Using Deep Learning
## 4 Performance Evaluation
### Model Performance
#### 4.2.2 Damage-Level Prediction Performance  
Three damage-level prediction models are assessed herein (Table 1). Figure 9 first demonstrates the damage-level prediction results using the same four sample inputs as in Figure 8 using the damage prediction model \(M_{DL}(FRC,RN)\). As shown earlier, the bounding boxes tend to envelop the buildings with different damage features. It is interesting to note that in Figure 9d, the damage prediction model chooses a much different bounding box and reports a correct damage level. In contrast, the underlying bounding-box emphasizes tsunami hazard features, and an erroneous damage level is given as 'ground-truth'. This implies the subjective variability introduced by human annotators. The performance measurements are reported in terms of the confusion matrices and scalar measures in Tables 4 and 5, respectively. The most significant observation is that damage-level prediction performance decreases considerably compared to the hazard-type prediction over the testing data. In terms of both the \(AUC\), \(F_{1}\) score, the \(AA\)'s, and \(OA\)'s, none of the performance measurement exceeds 0.9. Nonetheless, the model with the highest performance values is found in \(M_{DL}(FRC,RN)\), which shows moderate performance with an overall accuracy of 62.6% and \(AUC\) and \(F_{1}\) scores both greater than 0.5 at all damage-level predictions, albeit an alarming accuracy at predicting moderate-level damage. The two other damage-level prediction models, \(M_{DL}(FRC,ZF)\) and \(M_{DL}(SSD,RN)\), manifest unsatisfactory prediction performance. This implies that the'strong' Faster R-CNN model with a'moderate' ZF feature extractor or the 'normal' SSD model with a'strong' Resnet feature extractor cannot sufficiently discriminate damage-level as human experts can do. The model \(M_{DL}(FRC,RN)\) with a strong feature extractor and a strong region-proposal model can correctly detect damage-level. If one scrutinizes all the model performance measurements, even with two non-satisfactory models, it is evident that predicting moderate-level damage poses to be the most challenging one. This aspect, as well as the overall moderate performance, is further discussed later. Figure 8: Performance of \(M_{HT}(FRC,RN)\) in terms of: (**a**) Precision-Recall Curve; (**b**) ROC curve. The ROC and PRC plots in Figure 10 further illustrate the moderate predictive capacity of the model \(M_{DL}(FRC,RN)\). Compared to the ROC and PRC plots for the best hazard-type model \(M_{HT}(FRC,RN)\), it is seen that both predictive capacity and robustness degrade. Moreover, both graphical analytics show a relatively high capacity to predict \begin{table}
\begin{tabular}{c c c c c c c c c c c c c} \hline \hline \multirow{2}{*}{**ActualPredict**} & \multicolumn{3}{c}{\(M_{DL}(FRC,ZF)\)} & \multicolumn{3}{c}{\(M_{DL}(FRC,RN)\)} & \multicolumn{3}{c}{\(M_{DL}(SSD,RN)\)} \\ \cline{2-11}  & **Minor** & **Moderate** & **Major** & **Total** & **Minor** & **Moderate** & **Major** & **Total** & **Minor** & **Moderate** & **Major** & **Total** \\ \hline Minor & 27 & 24 & 10 & 61 & 46 & 7 & 8 & 61 & 29 & 8 & 24 & 61 \\ Moderate & 23 & 57 & 33 & 113 & 19 & 55 & 39 & 113 & 24 & 21 & 68 & 113 \\ Major & 10 & 34 & 36 & 80 & 4 & 18 & 58 & 80 & 13 & 15 & 52 & 80 \\ \hline Total & 60 & 115 & 79 & 254 & 69 & 80 & 105 & 254 & 66 & 44 & 144 & 254 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Confusion Matrix of damage-level Models.