Twitter Sentiment Analysis using NLTK, Python | by Mohamed Afham | Towards Data Science

Natural Language Processing (NLP) is a unique subset of Machine Learning which cares about the real life unstructured data. Although computers cannot identify and process the string inputs, the libraries like NLTK, TextBlob and many others found a way to process string mathematically. Twitter is a platform where most of the people express their feelings towards the current context. As humans, we can guess the sentiment of a sentence whether it is positive or negative. Similarly, in this article I’m going to show you how to train and develop a simple Twitter Sentiment Analysis supervised learning model using python and NLP libraries.
Let’s start working by importing the required libraries for this project.
Since it is a supervised learning task we are provided with a training data set which consists of Tweets labeled with “1” or “0” and a test data set without labels. The training and test data sets can be found here.
label “0”: Positive Sentiment
label “1”: Negative Sentiment
Now we will read the data with pandas.
train_tweets = pd.read_csv('train_tweets.csv')test_tweets = pd.read_csv('test_tweets.csv')
It is necessary to do a data analysis to machine learning problem regardless of the domain. Let’s do some analysis to get some insights.
sns.barplot('label','length',data = train_tweets,palette='PRGn')
sns.countplot(x= 'label',data = train_tweets)
The above two graphs tell us that the given data is an imbalanced one with very less amount of “1” labels and the length of the tweet doesn’t play a major role in classification. Dealing with imbalanced data is a separate section and we will try to produce an optimal model for the existing data sets.
The given data sets are comprised of very much unstructured tweets which should be preprocessed to make an NLP model. In this project, we tried out the following techniques of preprocessing the raw data. But the preprocessing techniques is not limited.
Removal of punctuations.
Removal of commonly used words (stopwords).
Normalization of words.
Punctuations will be always a disturbance in NLP specially hashtags and “@” play a major role in tweets. TextBlob’s word extraction feature from a sentence removes punctuations in an optimal level. The left out punctuations and other unusual notations will be removed in the upcoming preprocessing techniques.
In an NLP task the stopwords (most common words e.g: is, are, have) do not make sense in learning because they don’t have connections with sentiments. So removing them saves the computational power as well as increases the accuracy of the model.
All the unusual symbols and the numerical values were removed and returned a pure list with words as shown above. But still we may encounter multiple representations of the same word.(e.g: play, plays, played, playing) Even though the words are different they bring us the same meaning as the normal word “play”. So we need to do Lexicon Normalization approach to solve this issue. NLTK’s built-in WordNetLemmatizer does this requirement.
Now we have done our text preprocessing part and we will move onto the Vectorization and Model Selection.
Before we let our data to train we have to numerically represent the preprocessed data. The well-known techniques for vectorization of words in Natural Language Processing are: CountVectorization and Tf-IDF transformation. Let’s dive a bit into the theoretical background of those vectorization techniques.
CountVectorization generates a sparse matrix representing all the words in the document. A demonstration of Count Vectorization is given below:
Tf(d,t) (Term frequency) is defined as the number of occurrence of the term t in document d.
Idf(t) (Inverse document of frequency) is defined as log(D/t), where D: Total number of documents and t: Number of documents with the term.
What Tf-Idf transformer does is returns the product of Tf and Idf which is the Tf-Idf weight of the term.
So, we have now vectorized our sting data to numerical values in order to feed it to a machine learning algorithm. We choose naive bayes classifier for this binary classification since it is the most common algorithm used in NLP. Moreover, we use machine learning pipeline technique which is a built-in function of scikit-learn to pre-define a workflow of algorithm. This saves much time and computational power.
Since we decided to select the technique of the Natural Language Processing, we have to validate it with the existing training data set before applying to the test data set. We will use the conventional train_test_split technique to split the training data set with the test size of 0.2 and let our pipeline model be validated on the split data sets. Once we are satisfied with the validation accuracy, we can go forward to apply the obtained model to the test data set. Accuracy is measured using the built-in function of scikit-learn, confusion matrix and classification report.
An accuracy of 0.93837 is obtained for our simple pipeline model. Note that there are chances to improve this accuracy by tuning parameters using GridSearchCV and other preprocessing techniques.
Hope this article gave you a basic idea of sentiment analysis with NLTK and Python. The complete notebook for this project is available here.
Here are some useful links to get started with the libraries for Natural Language Processing we used in doing this project:
NLTK Book : A Complete guide for analyzing texts
TextBlob processing
Working with text data in scikit-learn
Sentiment Analysis Guide by MonkeyLearn