# DisKnow: A Social-Driven Disaster Support Knowledge Extraction System
## 3 Proposed Approach  
DisKnow was jointly developed at Iscte--Instituto Universitario de Lisboa and INOV INESC Inovacao. It is a system that retrieves tweets in real-time, filtering noisy data to better its performance and reliability. It then processes these tweets, using a language independent approach, and translates them to English, subsequently applying deep learning techniques to classify whether a tweet refers to real disasters or not. For relevant tweets, the extracted information is connected and related as different types of nodes, that are represented in the knowledge base, for a noise-free concise representation available to both population and decision makers. This knowledge is constantly being expanded and updated, thus achieving a desired level of adaptability to real threat scenarios, which are also continuously evolving, and require swift and informed responses. The knowledge base consists of a dynamic knowledge graph [39]. This allows us to structure entities as a knowledge graph in which new nodes and relationships are constantly being added as new context is detected, to expand and upgrade its knowledge in real-time. This knowledge is disaster-centered, as disaster occurrences are represented by specific nodes which then connect to locations, dates, people, organizations, etc. In addition to having several node types, such as the disasters and locations, each node also has several attributes. This allows for flexibility when dealing with various types of data associated with entities. As for its management, our knowledge base usesthe Neo4j ([[https://neo4j.com/](https://neo4j.com/)]([https://neo4j.com/](https://neo4j.com/))) graph database management system. The interactions with it are done through in-house developed custom methods, using a mixture of the Cypher query language, and the Py2 neo ([https://py2 neo.org/v4/](https://py2 neo.org/v4/)) toolkit. An overview on DisKnow's architecture is presented in Figure 1.  
1. **Tweet Extraction and Filtering:** The first step of our pipeline is the extraction of tweets and their related information, from the social media Twitter. To avoid overloading the system, these tweets are gathered through periodic API calls with several filters on them. These filters include the tweets' language and geolocation, which is done by defining a center and a radius, and calculating, for each tweet, the geodesic distance between the tweet's location and the defined center. Both the tweet's location and defined center are geocoded using data from OpenStreetMap [40], allowing for the computation of the mentioned geodesic distance. Relevant tweet information, such as each tweet's posting datetime and full text is also saved for further use. 2. **Tweet Preprocessing:** This step focuses on cleaning the textual content that is specific to tweets. As we have previously mentioned, one of the challenges of using social media information, is the huge amount of noisy data it pertains. This processing pipeline uses hand-made techniques, such as regex rules, that aim to take care of specific tweet noise, such as retweeting information, hyperlinks, user mentions, and hashtags. Firstly, any information regarding retweeting, hyperlinks, and user mentions is removed, and secondly hashtag symbols are removed and their content is split by upper-case starting words. 3. **Translating:** Translation is a natural step here, as DisKnow is being implemented as a component of a European project (grant agreement n833088) which aims to be put into effect in several countries. With this in mind, and to be able to use state-of-the-art models, which mostly exist for English, the input language is firstly detected, and then the cleaned text is translated to English. This is done using Google's Neural Machine Translation (GNMT) [41], which consists in a deep LSTM using both residual and attention connections. This model has achieved a BLEU [42] score of 38.95 in the commonly evaluated en-fr translation task, with a vocabulary size of around  
Figure 1: DisKnow’s process pipeline. 32,000 words.