# DisKnow: A Social-Driven Disaster Support Knowledge Extraction System
## 4 Evaluation
### Disaster Extraction Algorithms  
To compare our models with the current state-of-the-art literature on tweet disaster classification, we have divided this information in the two aforementioned classes: Event relatedness classification and Event type classification. We have then selected the best results of each relevant publication, being essential to highlight that all of these have been published between 2018 and 2019. The compared models use different algorithms and implementation techniques, but all of them use subsets of the same datasets--crisisLex--to train.  
Although not all the compared studies have shown their accuracy, precision, or recall, we have decided to also include these measures, as they may prove to be interesting in comparison with future works. To compare with the state-of-the-art however, as it is the most common measure for text classification, we use the F1-Score. This measure considers both precision and recall, and is often preferred in the field of information retrieval, as it offers a better estimate of wrongly classified cases, when compared to accuracy.  
\[\text{F1-score}=2\times\frac{\text{Precision}\times\text{Recall}}{\text{ Precision}+\text{Recall}} \tag{2}\]  
Our models have been trained using a subset of close to 20,000 labeled tweets. These tweets were balanced for the relatedness classification task, having around 54% related tweets and the remaining 46% unrelated. For the second task, however, there was a need to use under-sampling techniques to balance major classes--as our models were greatly overfitting those same classes--ending up with close to 5000 tweets.  
DisKnow's CNN related classifier has fared exceptionally well in this task. We have achieved an increase of almost 10 p.p of F1-Score when compared to the current state-of-the-art [36], with a test F1-Score of 92. This improvement can be explained by the usage of Twitter-specific embeddings, and variations in the model's architecture, such as pre-processing methods--which change the input and reduce the noise the network is exposed to--and the neural network architecture as well. Detailed results are shown in Table 1.  
As for the second task, disaster type classification, DisKnow has also achieved good results, although not as good as the state-of-the-art. A deep investigation on these results led us to believe  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Test Accuracy** & **Test Precision** & **Test Recall** & **Test F1-Score** \\ \hline CNN [36] & - & - & - & 0.838 \\ LSTM [37] & 0.748 & - & - & 0.751 \\ CNN—DisKnow & 0.903 & 0.921 & 91.9 & 0.920 \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparative results of the event relatedness classification task.  
that our type-classification CNN has a slight overfit to the flood class, sometimes misclassifying unrelated tweets. This problem is, however, fixed with the lexicon-based model that was presented in the proposed approach, although not reflected in these results, as to correctly compare the models. Despite this, further improvements may be needed in the implementation of this model. Detailed results are shown in Table 2.