# A Sentiment-Aware Contextual Model for Real-Time Disaster Prediction Using Twitter Data
## 4 Experiments
### Error Analysis  
Table 5 shows ten samples, including five positive and five negative ones, which are misclassified by the proposed SentiBERT-BiLSTM-CNN model. In this subsection, \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Precision** & **Recall** & **F1 Score** \\ \hline CNN & 0.8064 & 0.8086 & 0.8025 \\ BiLSTM & 0.8571 & 0.8405 & 0.8487 \\ SentiBERT & 0.8668 & 0.8712 & 0.8690 \\ BiLSTM-CNN & 0.8674 & 0.8523 & 0.8598 \\ word2 vec-BiLSTM-CNN & 0.8831 & 0.8767 & 0.8799 \\ fastText-BiLSTM-CNN & 0.8935 & 0.8736 & 0.8834 \\ BERT-BiLSTM-CNN & 0.9118 & 0.9187 & 0.9152 \\ BERT\({}_{\text{hyb}}\) & **0.9413** & 0.8992 & 0.9198 \\ SentiBERT-BiLSTM-CNN & 0.9305 & **0.9271** & **0.9275** \\ \hline \hline \end{tabular}
\end{table} 
Table 4: A performance comparison of models. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Loss Function** & **Epochs** & **Batch Size** & **Precision** & **Recall** & **F1 Score** \\ \hline \(L_{\text{BCE}}\) & 10 & 32 & 0.8987 & 0.8932 & 0.8959 \\ \(L_{\text{FL}}\) & 10 & 32 & 0.9029 & 0.9135 & 0.9082 \\ \(L_{\text{mix}}\) & 10 & 32 & **0.9305** & **0.9271** & **0.9275** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance of SentiBERT-BiLSTM-CNN under different loss function settings. we provide an analysis of these mistakes that may shed light on further improvement of our model. * For the five samples that are marked as disaster Tweets (i.e., samples one through five), none of them are describing a common sense disaster: sample 1 seems to state a personal accident; sample 2 talks about US dollar crisis which may indicate inflation given its context; in sample 3, the phrase \"batting collapse\" refers to a significant failure of the batting team in a sports game; sample 4 is the closest to a real disaster, but the word \"simulate\" simply reverses the semantic meaning; sample 5 does mention a disaster \"Catastrophic Man-Made Global Warming\", but the user simply expresses his/her opinion against it. Our observation is that the process of manual annotation could introduce some noises that would affect the modeling training. From another perspective, the noises help build more robust classifiers and potentially reduce overfitting. * For the five negative samples (6-10), we also observe possible cases of mislabeled samples: sample 6 clearly reports a fire accident with the phrase \"burning buildings\" but was not labeled as a disaster Tweet; sample 7 states a serious traffic accident; sample 8 mentions bio-disaster with the phrase \"infectious diseases and bioterrorism\"; sample 9 has only three words, and it is hard to tell its class without more context, although the word \"bombed\" is in the Tweet; sample 10 reflects a person's suicide intent, which could have been marked as a positive case. We need to clarify that these misclassified samples presented in the table are randomly selected from all error predictions. It can be seen that the length limit of Tweets presents pros and cons for training a classifier. The bright side is that users are forced to use short and direct words to express an opinion, and the downside is that some short Tweets are hard to interpret due to the lack of more context information, which is the main challenge for training an accurate model.