# DisKnow: A Social-Driven Disaster Support Knowledge Extraction System
## 3 Proposed Approach
It is also reported to reduce translation errors by more than 60%, when compared to its competition, the Phrase-based Machine Translation (PBMT) models [41]. 4. **Text Preprocessing:** Preprocessing is an essential step for natural language algorithms. Although activities like Named Entity Recognition (NER) benefit from inputting entire texts with punctuation and the original formatting, most algorithms benefit from further processed text. This step focuses on additionally preparing tweets for Disaster Extraction. It starts by tokenizing the tweets using Natural Language Toolkit ([[https://www.nltk.org/](https://www.nltk.org/)]([https://www.nltk.org/](https://www.nltk.org/))) (NLTK) algorithms, and proceeds to using regex rules to lowercase all the tokenized words. Lastly, English stop words are removed, as to lessen the noise inputted to the following steps. Although lemmatization or stemming are also frequently used, in our case we found that our Disaster Extraction algorithm slightly benefited from not using them. 5. **Named Entity Recognition:** NER is used as a tool to extract entities from the aforementioned tweets. These entities can be person names, companies, geographical locations, times, dates, etc. Their role is to complement a detected threat, in order to give users more insight. For this purpose, our system makes use of Spacy ([[https://spacy.io/](https://spacy.io/)]([https://spacy.io/](https://spacy.io/))) NER pipeline. As this pipeline is open, we are able to add new layers to it, to better embrace specific disaster detection characteristics, such as detailed locations and local businesses. 6. **Disaster Extraction:** The next step is, without any doubt, one of the most critical steps of our system. It consists of two sequential tasks: Firstly, classifying tweet relatedness to disasters, that is, classifying if a tweet's content is referring to a disaster event. Secondly, classifying the disaster type a tweet is referring to. To accomplish both of these tasks we have implemented two distinct convolutional neural networks (CNNs). The first CNN takes care of the first task, and if it classifies the tweet as being related to disasters, the second CNN then classifies which disaster it is, out of a predefined set of disasters it has been trained with, which are fire, flood, earthquake, explosion, and none. This last category exists as a double-check, seeing that the first CNN can detect disaster events that are not modeled by the second CNN. The second neural network outputs a list of confidence levels regarding all disaster classes. In the event of the second CNN classifying, a tweet with close confidence levels, \(\text{Conf1}-\text{Conf2}<\text{ConfMargin}\), where Conf1 refers to the highest confidence the CNN outputs, Conf2 refers to the second highest confidence, and Conf2 refers to the minimum margin set in the model, a weighted check is triggered. This check consists of adding to each of these confidence levels the confidence of a lexicon-based model. This model involves the preprocessing of the message with the use of tokenization and stemming techniques, and then comparing its result with predefined lexicons for each type of disaster. The results, LexCount, then go through a min-max normalization:  
\[\text{LexCount}^{\prime}=\frac{\text{LexCount}-\text{min}(\text{LexCount})}{ \text{max}(\text{LexCount})-\text{min}(\text{LexCount})} \tag{1}\]  
Lastly, a weighted ponderation of these values is added to the output confidence of the neural network, allowing for a better classification of the disaster a tweet refers to. This component's pipeline is also represented in Figure 2.  
The data used to train both networks were obtained as a subset of two CrisisLex [43] disaster datasets:  
Figure 2: Disaster extraction component pipeline. 1. CrisisLexT6 ([[https://crisislex.org/data-collections.html#CrisisLexT6](https://crisislex.org/data-collections.html#CrisisLexT6)]([https://crisislex.org/data-collections.html#CrisisLexT6](https://crisislex.org/data-collections.html#CrisisLexT6)));
2.