# DisKnow: A Social-Driven Disaster Support Knowledge Extraction System
## 4 Evaluation
### Top-Down Demonstration  
Our top-down demonstration consists in a simulation that was led in the municipality of Barreiro, a Portuguese city nearby Lisbon. This scenario aims to show what DisKnow's normal operation would be like, in a real scenario, from the moment it retrieves tweets, to the representation of the extracted information, in the Knowledge graph. As a result of the sheer impossibility of creating a large scale disaster for testing purposes, we had to resort to a simulated demonstration. Despite its smaller scale, this demonstration aims to show the contribution of DisKnow in disaster scenarios, automatically providing filtered and summed-up information to both local relevant authorities and population. In this demonstration, we have posted six relevant tweets at different moments. One hundred and thirty-six tweets were gathered by DisKnow. These tweets included the six tweets that we created and 130 other unknown tweets. Out of all these tweets, only the original six have successfully had impact in DisKnow's knowledge base, the remaining having been filtered through the system's pipeline, by either the Disaster Extraction or the Threat Object Generation component. Table 3 shows the amount of tweets outputted by each filtering component of the system. The simulation starts with a fire in the middle of a local factory called FISIPE. The flames are seen by three citizens that were passing by, and each posts a different tweet reporting the occurrence, using different levels of expressiveness. A few minutes later, someone reports that the fire has also spread to the nearby Vodafone store. Eventually, local news report tweets about the factory director having got a third-degree burn from the strong flames. About ten minutes after the fire started, a citizen also reports that the local Remax shop has been flooded at 12:30, and two of its workers have been injured. Figures 4-6 lay out the knowledge base on different moments of the simulation. Out of the initial three detections, two were written in Portuguese, and one in French. The follow-up tweets were all sent in Portuguese, and were written with different styles, to encompass what a real scenario would be like. Excluding all unrelated tweets, and keeping the order described in the scenario, the tweets detected by DisKnow were:  
1. **Estao chamas no telhado da FISIPE Barreiro! **
2. **Au secours! Fisipe brule a Barreiro**
3. **#Chamas #FISIPE fogo no Barreiro! ** \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Test Accuracy** & **Test Precision** & **Test Recall** & **Test F1-Score** \\ \hline SVM [36] & - & - & - & 0.997 \\ CNN [38] & 0.930 & - & - & 0.928 \\ CNN—DisKnow & 0.884 & 0.888 & 0.876 & 0.882 \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Comparative results of the event type classification task. \begin{table}
\begin{tabular}{c c} \hline \hline
**Component** & **Number of Tweets** \\ \hline Tweet Extraction and Filtering & 136 \\ Disaster Extraction (First CNN) & 9 \\ Disaster Extraction (Second CNN) & 7 \\ Threat Object Generation & 6 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Information filtering throughout the system. The impact of the fourth and fifth tweet is demonstrated in Figure 5. These tweets also referred to the same threat, but added new information, which was the fire spreading to a nearby store, and the director of the factory being injured. DisKnow correctly detected that this was an already reported occurrence, and proceeded to extract new information, which was then represented in the knowledge graph, by creating the new person and organization nodes and connecting them to the ongoing fire threat, with the AFFECTED relationship. Lastly, in Figure 6 the result of the sixth tweet is included. This tweet refers to a completely different threat that was reported. Seeing that the citizen specified a start time for the flood, DisKnow used that value, instead of the time the tweet was published at.