This Is How Twitter Sees The World : Sentiment Analysis Part One | by Ronald Wahome | Towards Data Science

Twitter is an online social network with over 330 million active monthly users as of February 2018. Users on twitter create short messages called tweets to be shared with other twitter users who interact by retweeting and responding. Twitter employs a message size restriction of 280 characters or less which forces the users to stay focused on the message they wish to disseminate. This very characteristic makes messages on twitter very good candidates for the Machine Learning (ML) task of sentiment analysis. Sentiment Analysis falls under Natural Language Processing (NLP) which is a branch of ML that deals with how computers process and analyze human language.
The training data was obtained from Sentiment140 and is made up of about 1.6 million random tweets with corresponding binary labels. 0 for Negative sentiment and 1 for Positive sentiment. In this blog post, we’ll use a Naive Bayes Classifier to learn the correct labels from this training set and do a binary classification. In a nut shell, the Naive Bayes theorem calculates the probability of a certain event happening based on the joint probabilistic distributions of certain other events. I downloaded the test dataset using twitter’s API and will be use to test the model’s real world performance. Full documentation and terms of the API are available at developer.twitter.com/en/docs.
In the first section of this blog post we’ll go through standard preprocessing steps performed on text data for a sentiment analysis ML task. The second section is covered in a subsequent blog post where we combine the preprocessing into one step within a ML pipeline. This first post goes through the trouble of explaining what is happening under the hood when we use a pipeline to handle preprocessing.
We’ll use Jupyter Notebook with Python for our workflow.
Import the python libraries used in this project.
Import the data to a pandas dataframe and do some exploratory data analysis.
# you can see the full list of imports on GitHub!# Machine Learning importsimport nltkfrom sklearn.pipeline import Pipelinefrom sklearn.model_selection import train_test_splitfrom sklearn.metrics import classification_report, confusion_matrix, accuracy_scorefrom sklearn.naive_bayes import MultinomialNBfrom sklearn.model_selection import KFold, cross_val_scorefrom sklearn.ensemble import RandomForestClassifierfrom sklearn.feature_extraction.text import CountVectorizerfrom sklearn.feature_extraction.text import TfidfTransformerfrom sklearn.model_selection import GridSearchCVfrom sklearn.externals import joblibfrom nltk.corpus import stopwordsfrom nltk.tokenize import TweetTokenizerfrom nltk.stem.wordnet import WordNetLemmatizerLoad training dataset to Pandas and preview the top rows.
# load train datadata = pd.read_csv('Sentiment Analysis Dataset.csv',                   error_bad_lines=False)data.columns = ['id','label','source','text']data.head(2)# get text and matching label columnsdata = data.drop(['id','source'],axis=1)data.head(10)
We can observe that the data is indeed from tweet messages posted on twitter.
The labels and the text do not seem to be in any listed order. This can be a problem if data is not randomly distributed as it can introduce biases to a learning model. In any case, we are going to use the Scikit Learn library which has a function to split our training and testing data and shuffle the data at the same time.
Shuffling data reduces variance and makes sure our model can generalize better on the data and do less overfitting. We want to make sure the train and test dataset are representative of the overall distribution of the data.
To that note, we also want to check the label frequency distribution in the data which is done below.
Another observation we can make is that the text contains varying formats. Some words contain mixed case letters which need to be normalized to their base word. e.g.: ‘CRy’ changed to ‘cry’.
Leaving words with first letter capitalized can be experimented with as they may hold a different feature space like the name of a person or country etc.
Explore distribution of label types.
# check the number of positive vs. negative tagged sentencespositives = data['label'][data.label == 0]negatives = data['label'][data.label == 1]print('number of positve tagged sentences is:  {}'.format(len(positives)))print('number of negative tagged sentences is: {}'.format(len(negatives)))print('total length of the data is:            {}'.format(data.shape[0]))
Considering the size of the dataset, the labels seem to be ‘about’ evenly distributed at 788435 vs. 790177 for positive and negative respectively.
Next, we want to see the number of words contained in every sentence so I created a function to extract this information and appended it to a column next to the text column. Below is a sample output.
# get a word count per sentence columndef word_count(sentence):    return len(sentence.split())    data['word count'] = data['text'].apply(word_count)data.head(3)
# plot word count distribution for both positive and negative sentimentsx = data['word count'][data.label == 1]y = data['word count'][data.label == 0]plt.figure(figsize=(12,6))plt.xlim(0,45)plt.xlabel('word count')plt.ylabel('frequency')g = plt.hist([x, y], color=['r','b'], alpha=0.5, label=['positive','negative'])plt.legend(loc='upper right')
From the graph above, most sentences fall between 5–10 words but it’s fair to say that majority of text on twitter falls between 1 and 25 words. This is no wonder considering that twitter has a limit of how many characters one can use in a message. 280 characters is the limit at the time of this writing.
In all, it looks like 1–20 words covers more than 80% of all sentences which makes this dataset set a good training candidate.
There are more positive sentences with 5 words or less than there are negative ones which does not seem like a big enough difference to cause any concern at the moment.
# get most common words in training datasetall_words = []for line in list(data['text']):    words = line.split()    for word in words:        all_words.append(word.lower())        Counter(all_words).most_common(10)
Output below:
[('i', 741876), ('to', 556149), ('the', 516654), ('a', 374024), ('my', 309966), ('and', 294805), ('you', 236109), ('is', 229444), ('for', 212852), ('in', 209009)]
In the cell above we extracted the most common words in the dataset and listed the top ten.
Perhaps to no surprise we encounter words like i, and and is as they are very highly used in human expressions. These kind of words usually appear equally in both negative and positive oriented expressions and as such they bring very little information that can be incorporated in the model so we will have to get rid of them down the road.
In the text preprocessing steps later on we will learn how to deal with these common words that don’t add much to the feature space.
Below is a code to output a graph showing the frequency of the first 25 words.
# plot word frequency distribution of first few wordsplt.figure(figsize=(12,5))plt.title('Top 25 most common words')plt.xticks(fontsize=13, rotation=90)fd = nltk.FreqDist(all_words)fd.plot(25,cumulative=False)# log-log plotword_counts = sorted(Counter(all_words).values(), reverse=True)plt.figure(figsize=(12,5))plt.loglog(word_counts, linestyle='-', linewidth=1.5)plt.ylabel("Freq")plt.xlabel("Word Rank")plt.title('log-log plot of words frequency')
I also created a log-log plot for the words frequency which is similar to the previous frequency graph but includes all words and is plotted on a base 10 logarithmic scale which helps us visualize the rapidly diminishing frequency of words as their rank drops.
The word distribution present in this data dictionary is a very common phenomenon in large samples of words as shown by Zipf’s law where the most frequent word will occur about twice as often as the second most frequent word, three times as often as the third most frequent word, etc.
It would be interesting to see if this holds true after we remove words like i, and and is from the observation above.
After training the model we make sentiment predictions on the raw twitter data but before we can do that, we need to download and do some basic data cleaning. You download tweets off the twitter RESTful API by keyword and get a stream of historical tweets back along with the metadata. I used Tweepy which is a python wrapper library for the twitter API that gives you more control on how you query the API. You can see the code on GitHub.
The keyword used to download historical tweets was “Paul Ryan” who is the speaker of the US House of Representative at the time of writing. My choice for this keyword is arbitrary but I feel is good enough to return some polarizing tweets. This is important because the tweets themselves can act as their own benchmark. One of the obstacles of NLP is evaluating a model’s performance considering that language is relative and always changing.
Import and preview raw data.
# create column namescol_names=['date','user_loc','followers','friends','message','bbox_coords',\           'full_name','country','country_code','place_type']# read csvdf_twtr = pd.read_csv('paul_ryan_twitter.csv', names=col_names)# check headdf_twtr.head()
We’ll keep the location data and use it later to map out our results on a geographic map.
We want to remove any unnecessary qualities in the data which would make the trained model a poor generalizer.
Text preprocessing involves many things like removing emojis, properly formatting the text to remove extra spaces or any other information in the text that we don’t believe would add information to our model. We’ll see some examples below.
We also have to make sure that the information we pass the model is in a format that computers can understand. We’ll also go through some of these steps below.
After this pre-processing step, our data should be ready to use for a machine learning classification task.
NOTE : Whatever text preprocessing we do on the train data must also be done on the test and our raw data.
This step is one that we could spend a lot of time on but the goal is always to find the best balance.
The bulk of the work in NLP is done on feature engineering. For now we will get rid of the links and emoji characters. It is important to mention that we can use them as features but for now we’re just building a basic model.
I created a function that uses regex to do bulk formatting for every tweet in the dataset. Sample output is included under the code.
# helper function to clean tweetsdef processTweet(tweet):    # Remove HTML special entities (e.g. &amp;)    tweet = re.sub(r'\&\w*;', '', tweet)    #Convert @username to AT_USER    tweet = re.sub('@[^\s]+','',tweet)    # Remove tickers    tweet = re.sub(r'\$\w*', '', tweet)    # To lowercase    tweet = tweet.lower()    # Remove hyperlinks    tweet = re.sub(r'https?:\/\/.*\/\w*', '', tweet)    # Remove hashtags    tweet = re.sub(r'#\w*', '', tweet)    # Remove Punctuation and split 's, 't, 've with a space for filter    tweet = re.sub(r'[' + punctuation.replace('@', '') + ']+', ' ', tweet)    # Remove words with 2 or fewer letters    tweet = re.sub(r'\b\w{1,2}\b', '', tweet)    # Remove whitespace (including new line characters)    tweet = re.sub(r'\s\s+', ' ', tweet)    # Remove single space remaining at the front of the tweet.    tweet = tweet.lstrip(' ')     # Remove characters beyond Basic Multilingual Plane (BMP) of Unicode:    tweet = ''.join(c for c in tweet if c <= '\uFFFF')     return tweet# ______________________________________________________________# clean dataframe's text columndf_paulry['message'] = df_paulry['message'].apply(processTweet)# preview some cleaned tweetsdf_paulry['message'].head()
Also note that you cannot have a perfect set of clean up steps as some cleanup steps will ultimately introduce some flaws in the data.
Below is a before and after example where ‘re’ has been removed from re-election which alters the meaning of the word and perhaps the user’s intended meaning. The word won’t also changes to won which obviously has a totally different meaning.
We drop duplicate tweets as they bring no new information to the dataset and are also computationally inefficient. We also visualize the most common words in the corpus.
# most common words in twitter datasetall_words = []for line in list(df_paulry['message']):    words = line.split()    for word in words:        all_words.append(word.lower())# plot word frequency distribution of first few wordsplt.figure(figsize=(12,5))plt.xticks(fontsize=13, rotation=90)fd = nltk.FreqDist(all_words)fd.plot(25,cumulative=False)# log-log of all words word_counts = sorted(Counter(all_words).values(), reverse=True)plt.figure(figsize=(12,5))plt.loglog(word_counts, linestyle='-', linewidth=1.5)plt.ylabel("Freq")plt.xlabel("Word Rank")
Our training data has now been transformed into a much leaner body of text that is much cleaner for feature extraction. As we mentioned before, we do have some words in the dataset that are common in natural human language but used in most sentence compositions that we would be better left off since they bring no useful features to our model.
These words are commonly known as stop-words in NLP and the NLTK library comes with a function that can filter out these words from the dataset. Below is the actual words contained in the stop_words list. We can also make our own special list of stop words to fit any unique case. For example if you are doing sentiment analysis on law documents, you would probably need a special set considering the law jargon contained is unique in nature.
# show stop words examples("i , me , my , myself , we , our , ours , ourselves , you , you're , you've , you'll , you'd , your , yours , yourself , yourselves , he , him , his , himself , she , her , hers , herself , it ")
After removing stop-words we split all the sentences in the dataset to get individual words (tokens) which is basically a list of words per sentence contained in the newly processed tweet. Now we can see that we have two new columns in the dataframe that contains these tokenized versions of a tweet.
A sample output is included below.
# tokenize helper functiondef text_process(raw_text):    """    Takes in a string of text, then performs the following:    1. Remove all punctuation    2. Remove all stopwords    3. Returns a list of the cleaned text    """    # Check characters to see if they are in punctuation    nopunc = [char for char in list(raw_text) if char not in string.punctuation]    # Join the characters again to form the string.    nopunc = ''.join(nopunc)        # Now just remove any stopwords    return [word for word in nopunc.lower().split() if word.lower() not in stopwords.words('english')]def remove_words(word_list):    remove = ['paul','ryan','...','“','”','’','...','ryan’']    return [w for w in word_list if w not in remove]# -------------------------------------------# tokenize message column and create a column for tokensdf_paulry = df_paulry.copy()df_paulry['tokens'] = df_paulry['message'].apply(text_process) # tokenize style 1df_paulry['no_pauls'] = df_paulry['tokens'].apply(remove_words) #tokenize style 2df_paulry.head()
There are additional normalization techniques like stemming and lemmatizing that we can try on our data but twitter messages are short by design and the above methods may not work so well because they essentially shorten words to their base words. e.g.: running to run.
For now we will stick with the current normalized state of our message data which we can now convert into a vector that can be fed into the appropriate ML algorithim.
I have also created a word cloud depicting the most common words in the entire twitter dataset after normalization.
We can see our keywords — paul & ryan — are obviously very prominent but some domain knowledge would be helpful in making sense of why some of the other words are there. It is twitter after all so you can always expect a wide range of emotions. Some words in the dataset do use very strong language.The word cloud also excludes the words paul and ryan as they are disproportionately common in the dataset.
# split sentences to get individual wordsall_words = []for line in df_paulry['no_pauls']: # try 'tokens'    all_words.extend(line)    # create a word frequency dictionarywordfreq = Counter(all_words)# draw a Word Cloud with word frequencieswordcloud = WordCloud(width=900,                      height=500,                      max_words=500,                      max_font_size=100,                      relative_scaling=0.5,                      colormap='Blues',                      normalize_plurals=True).generate_from_frequencies(wordfreq)plt.figure(figsize=(17,14))plt.imshow(wordcloud, interpolation='bilinear')plt.axis("off")plt.show()
We’ll convert each message which is represented by a list of tokens into a vector that a machine learning model can understand.
To do this we use the Bag Of Words model which is a three step process.
1.Count how many times does a word occur in each message (Known as term frequency)
2.Weigh the counts, so that frequent tokens get lower weight (inverse document frequency)
3.Normalize the vectors to unit length, to abstract from the original text length (L2 norm)
Each vector will have as many dimensions as there are unique words in the tweeter corpus.
We will first use SciKit Learn’s CountVectorizer function which converts a collection of text documents to a matrix of token counts.
Imagine this as a 2-D matrix where 1-D is the entire vocabulary contained in the messages and the other dimension is one column per tweet.
Since there are so many messages, we can expect a lot of zero counts for the presence of every word in the data but SciKit Learn will output a Sparse Matrix. Below is an example — code and output — of a vectorized sentence.
# vectorizebow_transformer = CountVectorizer(analyzer=text_process).fit(df_paulry['message'])# print total number of vocab wordsprint(len(bow_transformer.vocabulary_))# output6865# example of vectorized textsample_tweet = df_paulry['message'][111]print(sample_tweet)print('\n')# vector representationbow_sample = bow_transformer.transform([sample_tweet])print(bow_sample)print('\n')
# transform the entire DataFrame of messagesmessages_bow = bow_transformer.transform(df_paulry['message'])# check out the bag-of-words counts for the entire corpus as a large sparse matrixprint('Shape of Sparse Matrix: ', messages_bow.shape)print('Amount of Non-Zero occurences: ', messages_bow.nnz)
TF-IDF stands for term frequency-inverse document frequency, and the tf-idf weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus.
The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus. Variations of the tf-idf weighting scheme are often used by search engines as a central tool in scoring and ranking a document’s relevance given a user query.
One of the simplest ranking functions is computed by summing the tf-idf for each query term; many more sophisticated ranking functions are variants of this simple model.
Typically, the tf-idf weight is composed by two terms: the first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document; the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.
TF: Term Frequency, which measures how frequently a term occurs in a document. Since every document is different in length, it is possible that a term would appear much more times in long documents than shorter ones. Thus, the term frequency is often divided by the document length (aka. the total number of terms in the document) as a way of normalization:
TF(t) = (Number of times term t appears in a document) / (Total number of terms in the document).
IDF: Inverse Document Frequency, which measures how important a term is. While computing TF, all terms are considered equally important. However it is known that certain terms, such as “is”, “of”, and “that”, may appear a lot of times but have little importance. Thus we need to weigh down the frequent terms while scale up the rare ones, by computing the following:
IDF(t) = log_e(Total number of documents / Number of documents with term t in it).
See below for a simple example.
Example: Consider a document containing 100 words wherein the word cat appears 3 times. The term frequency (i.e., tf) for cat is then (3 / 100) = 0.03. Now, assume we have 10 million documents and the word cat appears in one thousand of these. Then, the inverse document frequency (i.e., idf) is calculated as log(10,000,000 / 1,000) = 4. Thus, the Tf-idf weight is the product of these quantities: 0.03 * 4 = 0.12.
# from sklearn.feature_extraction.text import TfidfTransformertfidf_transformer = TfidfTransformer().fit(messages_bow)tfidf_sample = tfidf_transformer.transform(bow_sample)print(tfidf_sample)
We’ll go ahead and check what is the IDF (inverse document frequency) of the word “trump” and of word “gop”?
# some IDF (inverse document frequency) exampleprint(tfidf_transformer.idf_[bow_transformer.vocabulary_['trump']])print(tfidf_transformer.idf_[bow_transformer.vocabulary_['gop']])
To transform our entire twitter bag-of-words corpus into TF-IDF corpus at once we’ll use the code below:
# to transform the entire bag-of-words corpusmessages_tfidf = tfidf_transformer.transform(messages_bow)print(messages_tfidf.shape)# Output(4164, 6865)
After preprocessing the data, we are now ready to pass it through a ML classification algorithim.
This ends the text preprocessing stages of our workflow but we will revisit these steps as one step in the pipeline in the next section continued here.