Binary Classification of Disaster Tweets | by Sohail Hosseini | Towards Data Science

In recent years, social media has gotten lots of attention for their potential use in spatial and temporal events. Twitter is one of them that has become an important communication channel in different situations, for instance, in times of emergency. The smartphones enable people to announce an emergency they see in real-time. Because of that, more agencies are interested in monitoring Twitter (e.g., disaster relief organizations). The idea behind this work is that we can take a large body of tweets and extract a useful summary that could have been useful in disaster situations.
The dataset could be downloaded from here. We have 7613 observations, and we are predicting whether a given tweet is about a real disaster or not. If so, predict a 1. If not, predict a 0. Features and their descriptions are as mentioned below.
id - an identifier for each tweet
text - the text of the tweet
location - the location the tweet was sent from (maybe NaN)
keyword - a particular keyword from the tweet (maybe NaN)
target - the output that tells if a tweet is about a real disaster (1) or not (0)
But how can we analyze these sorts of problems?
Let’s look at features included in dataset:
There are 7613 observations and 5 features in this dataset.
As it shows, the dataset consists of 7613 reviews. They are five different features available in this dataset, but we only work ‘text’ and ‘target’ columns as our input/output, respectively. Now, let’s look at features included in dataset:
There are a lot of missing values in keyword and location columns, but we do not need to be worried about them. To make our life easier, we would need to clean noises in the text feature. We will talk about it in the following sections.
If we visualize the number of disaster tweets to non-disaster ones shown below, it is clear that the dataset is not balanced.
To solve this issue, we are going to use stratified sampling, where the dataset is divided into homogeneous subgroups called strata, and the right number of instances is sampled from each stratum to guarantee that the test set is representative of the overall population. We can see that the train and test datasets both have almost 41% disaster tweets. The code for stratified sampling is provided below.
The train and test datasets are unbalanced. But there are several methods to deal with an unbalanced dataset as oversampling and undersampling. Fortunately, many of these applied algorithms here can solve this matter for you. There is a hyperparameter named, ‘class_weight’ used the values of the target to automatically adjust weights inversely proportional to class frequencies in the dataset.
Let’s look at the type of each feature with pandas .info() function, which also can give you a general overview of features. It can be helpful to know if there is any missing value for each column, so you will be able to deal with it effectively.
train.info()
Since we are only working with only two columns, i.e. (‘text’ and ‘target’), and there is no missing value, we do not need to remove any observation or use an imputation transformer for completing missing values.
Few tweets are shown below. Remember, we are going to use them to predict if there are disaster tweets or not. What actually can help us to remove redundant words, numbers, etc. that do not add any value to texts, for example, hashtags, https, numbers. So we are going to remove character digits and pieces of text that interfere with our text analysis.
With the following code, I have removed digits, non-ASCII characters, punctuations, https, and changed all texts to lower.
Below, you can see a few cleaned texts that we will use for text analysis.
Our goal here is to find a relation between input and output. I use the ‘cleaned’ feature as input and ‘target’ as output. So my first step is to convert tweets to vectors.
We need to convert tweets to a matrix of token counts to extract numerical features from text content. By doing it, a sparse representation of the counts is generated. It is very beneficial as you can imagine unique mapping words to vectors create the massive size of a matrix. CountVectorizer is used for this purpose, which can be imported as follows.
from sklearn.feature_extraction.text import CountVectorizer
There are many words like ‘the,’ ‘them,’ ‘are’ that do not have any effect on the meaning of context, which are called stop words. They are not informative and can be removed by selecting step_words=’english’ as a hyperparameter in CountVectorizer function.
The next step is to normalize the count matrix using tf-idf representation. The main reason behind normalizing frequencies instead of using the raw ones is to decrease the effect of tokens that occur several times in the text, and there’s not as informative as those presented a few times. For instance, the word “document” occurs a thousand times in a given corpus, and “awesome” happens two times. tf-idf works in this situation, like preprocessing data to change raw feature vectors into a representation that is more suitable for machine learning algorithms.
from sklearn.feature_extraction.text import TfidfTransformer
I used the pipeline function to do all steps together. Sequentially apply a list of transforms and a final estimator. Therefore, it starts asking CountVectorizer and Tfidf. You can have any number of transformers required for your work in Pipeline.
I have applied pipelines only on the ‘cleaned’ feature because the target is binary.
I have used 20% of the data for the test and the rest for training. There are a lot of hyperparameters involved here. So your job is like an architect to find the best values to maximize accuracy. Several techniques are beneficial, including Hyperopt, GridSearchCV, and RandomizedSearchCV. In this problem, I have selected hyperparameters randomly.
I have used twelve different supervised algorithms mentioned in the code snippet below. Area Under Curve (AUC) score and training time are used for comparison.
One by one, every supervised algorithm is trained on the trained dataset and then generalized on the test dataset. In the end, results are ordered based on the value of AUC. Training time for each method is also shown. I have used this kernel to write down the following classification code.
As we can see, the linear support vector machine has a maximum AUC score but not the minimum training time. Since we have not optimized hyperparameters, you can change them to trade off accuracy and speed.
SVM has not been trained in the fastest time. Although hyperparameters were not optimized, text cleaning can reduce the noise in text datasets in the form of stopwords, punctuations marks, and numbers, etc. In real-world problems, speed is more important than accuracy. So as the size of data is increased, we might want to focus on speed. Deep learning could also be used for comparison. Some algorithms also use GPUs like Lightgbm, so if you have a massive size of data, you might want to consider using that method.
All the code used in this article can be accessed from my GitHub. I look forward to hearing feedback or questions.