# Understanding Natural Disaster Scenes from Mobile Images Using Deep Learning
## 4 Performance Evaluation
### Model Performance
#### 4.2.1 Hazard-Type Prediction Performance  
Three hazard-type prediction models are assessed herein (Table 1). Figure 7 demonstrates four sample prediction results from \(M_{HT}(FRC,RN)\). In each predicted instance, both ground-truth and predicted information are annotated, including the bounding boxes, the class labels, and the prediction scores. Regarding the bounding-box prediction, first, it is observed that our strategy of emphasizing the spatial boundaries of damaged buildings is largely confirmed. In all cases, the bounding boxes tend to envelop the buildings. It is observed that for tornado scenes, as illustrated in Figure 7a,b, both bounding boxes and hazard-type are more accurately detected. For earthquake and tsunami scenes, instances exist that are challenging to differentiate for human analysts if one inspects Figure 7c,d. Nonetheless, the model largely has learned the salient differences in terms of geometric distinction of debris patterns. In addition, while outlining the bounding box in Figure 7d, it seems that the analyst emphasizes the salient region that can inform the tsunami hazard-type. Therefore, a smaller box is given. In terms of discriminating its damage-level, it is excessively small. Based on the testing data, the accuracy measurements are reported in two tables. Table 2 reports the confusion matrix for each hazard-type model. In Table 3, four accuracy measures are listed, including the AUC based on the ROC curve as a measure of model capacity, the \(F_{1}\) score as the primary accuracy measure, and the \(AA\) as a simple accuracy measure. These three measures are calculated as different class labels (the hazard-type of Tornado, Tsunami, and Earthquake). Then the overall accuracy measure, \(OA\), is given  
Figure 7: Hazard-type prediction using \(M_{HT}(FRC,RN)\) (red-bounding boxes from prediction; and blue boxes from testing data): (**a**) Tornado-wind scene (correct prediction with a score of 100.0%); (**b**) Tornado-wind scene (correct prediction with a score of 100.0%); (**c**) Earthquake-shaking scene (correct prediction with a score of 99.9%); (**d**) Tsunami-wave scene (correct prediction scene with a score of 90.18%). for each model. The following observations are summarized based on these performance measurements. First, the high \(AUC\) scores of both the FRC-based models signify their higher prediction capacities than the SSD model at all hazard types. In terms of the \(F_{1}\) scores, the FRC-based models again show much greater accuracy than the SSD model. If the \(F_{1}\) scores alone are compared, one may see that when the Resnet-50 is used as the feature extractor, slightly better classification accuracy is observed than the ZF-based model. The \(AA\) measure shows a consistent trend as the \(F_{1}\) score. On the other hand, when the SSD model \(M_{HT}(SSD,RN)\) is concerned, even with the more competitive feature extractor (ResNet-50), its accuracy drops significantly. Based on this evidence, it is argued that the use of Faster R-CNN as the basis for hazard-type prediction is superior to the SSD-based architecture. Between the two Faster R-CNN models, \(M_{HT}(FRC,ZF)\) and \(M_{HT}(FRC,RN)\), it can be seen that this hazard-type detector is more sensitive to the tornado disaster than to the earthquake disaster, and the last is the tsunami disaster. To comprehend this observation, the percentages of images of different hazard types in the learning datasets may provide some insight. It is found that the earthquake images are about 42% of the total samples, tornado images about 25%, and tsunami images around 33%. This indicates that the data is relatively not well-balanced but not severely imbalanced, implying that data imbalance does not sufficiently explain the lowest performance in tsunami disaster prediction.