Social Media Sentiment Analysis using Machine Learning : Part — II
By: Deepak Das
Tags: Machine Learning, Sentiment Analysis, Python, Naturallanguageprocessing, Data Science

Photo by Tengyart on Unsplash

Hello everyone, so let’s start right where we left off in Part — I.

In this post we will discuss how we can extract features from our textual dataset by using Bag-of-Words and TF-IDF. Then we will see how we can apply Machine Learning models using these features to predict whether a tweet falls into the Positive: ‘0’ or Negative: ‘1’ sentiment.

Note : In case you haven’t read the Part — I of this series do give it a read to get a better understanding of Part — II.

So, let’s start shall we ?

Extracting Features from cleaned Tweets

Bag-of-Words Features

Bag of Words is a method to extract features from text documents. These features can be used for training machine learning algorithms. It creates a vocabulary of all the unique words occurring in all the documents in the training set.

Consider a corpus (a collection of texts) called C of D documents {d1,d2…..dD} and N unique tokens extracted out of the corpus C. The N tokens (words) will form a list, and the size of the bag-of-words matrix M will be given by D X N. Each row in the matrix M contains the frequency of tokens in document D(i).

For example, if you have 2 documents-

D1: He is a lazy boy. She is also lazy.

D2: Smith is a lazy person.

First, it creates a vocabulary using unique words from all the documents.

[‘He’ , ’She’ , ’lazy’ , ‘boy’ , ‘Smith’ , ’person’]

As we can see in the above list we don’t consider “is” , “a” , “also” in this set because they don’t convey the necessary information required for the model.

Here, D=2, N=6

The matrix M of size 2 X 6 will be represented as:

The above table depicts the training features containing term frequencies of each word in each document. This is called bag-of-words approach since the number of occurrence and not sequence or order of words matters in this approach.

So,let’s apply this word embedding technique to our available dataset.

We have a package called CountVectorizer to perform this task.

OUTPUT :-

TF-IDF Features

TF-IDF stands for Term Frequency-Inverse Document Frequency, and the TF-IDF weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.

Typically, the TF-IDF weight is composed by two terms:

Term Frequency (TF) :

The first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document.

Example :-

Consider a document containing 100 words wherein the word cat appears 3 times.

The Term Frequency (TF) for cat is then (3 / 100) = 0.03

Inverse Document Frequency (IDF) :

The second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

Example :-

Assume we have 10 million documents and the word cat appears in one thousand of these.

Then, the Inverse Document Frequency (IDF) is calculated as

log(10,000,000 / 1,000) = 4.

TF-IDF Example :

Formula for finding the TF-IDF weight :-

From the above examples the Term Frequency is 0.03 and Inverse Document Frequency is 4.

Thus, the TF-IDF weight is the product of these quantities : 0.03 * 4 = 0.12.

CODE :-

Let us apply this technique to our dataset using Python.

We have a package available for this in Scikit-Learn known as TfidfVectorizer.

OUTPUT :-

These are the Word Embedding techniques which we have used on our dataset for feature extraction.

Let’s move on to the next step.

Splitting of our dataset into training and validation set.

Splitting our dataset into Training and Validation Set

From the above two techniques that is Bag-of-Words and TF-IDF we have extracted features from the tweets present in our dataset.

Now, we have one dataset with features from the Bag-of-Words model and another dataset with features from TF-IDF model.

First task is to split the dataset into training and validation set so that we can train and test our model before applying it to predict for unseen and unlabeled test data.

Using the features from Bag-of-Words for training set

train_bow = bow[:31962]



train_bow.todense()

OUTPUT :-

Using features from TF-IDF for training set

train_tfidf_matrix = tfidf_matrix[:31962]



train_tfidf_matrix.todense()

OUTPUT :-

Splitting the data into training and validation set

from sklearn.model_selection import train_test_split

Bag-of-Words Features

x_train_bow, x_valid_bow, y_train_bow, y_valid_bow = train_test_split(train_bow,train['label'],test_size=0.3,random_state=2)

TF-IDF features

x_train_tfidf, x_valid_tfidf, y_train_tfidf, y_valid_tfidf = train_test_split(train_tfidf_matrix,train['label'],test_size=0.3,random_state=17)

We are done with splitting our dataset into train and validation set.

Want to know more about the Bag-of-Words and TF-IDF models used for feature extraction. Do give a read to the following blog post for an in depth discussion.

Finally , we are here for the most awaited part of this series that is applying Machine Learning Models on our dataset.

Applying Machine Learning Models

The underlying problem we are going to solve comes under the Supervised Machine Learning category. So, let us have a brief discussion about this topic before moving on to apply different Machine Learning models on our dataset.

Supervised Machine Learning :-

The majority of practical machine learning uses supervised learning.

Supervised learning is where you have input variables (x) and an output variable (Y) and you use an algorithm to learn the mapping function from the input to the output.

Y = f(X)

The goal is to approximate the mapping function so well that when you have new input data (x) that you can predict the output variables (Y) for that data.

It is called supervised learning because the process of an algorithm learning from the training dataset can be thought of as a teacher supervising the learning process. We know the correct answers, the algorithm iteratively makes predictions on the training data and is corrected by the teacher. Learning stops when the algorithm achieves an acceptable level of performance.

Supervised learning problems can be further grouped into regression and classification problems.

Classification : A classification problem is when the output variable is a category, such as “red” or “blue” or “disease” and “no disease” or in our case “Positive” or “Negative”

: A classification problem is when the output variable is a category, such as or or and or in our case or Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.

Our problem comes under the classification category because we have to classify our results into either Positive or Negative class.

There is another category of Machine Learning algorithm called Unsupervised Machine Learning where you have an input data but no corresponding output variables. The goal for unsupervised learning is to model the underlying structure or distribution in the data in order to learn more about the data. But that is of no concern for us for this problem statement.

Moving on :-

So from the above splitting of dataset we see that we will use features from the Bag-of-Words and TF-IDF for our Machine Learning Models.

We generally use different models to see which best fits our dataset and then we use that model for predicting results on the test data.

Here we will use 3 different models

Logistic Regression

XGBoost

Decision Trees

and then we will compare their performance and choose the best possible model with the best possible feature extraction technique for predicting results on our test data.

Importing f1_score from sklearn

We will use F1 Score throughout to asses our model’s performance instead of accuracy. You will get to know why at the end of this article.

CODE :-

from sklearn.metrics import f1_score

Now, let’s move on to applying different models on our dataset from the features extracted by using Bag-of-Words and TF-IDF.

Logistic Regression

The first model we are going to use is Logistic Regression.

from sklearn.linear_model import LogisticRegression Log_Reg = LogisticRegression(random_state=0,solver='lbfgs')

Bag-of-Words Features

Fitting the Logistic Regression Model.

Log_Reg.fit(x_train_bow,y_train_bow)

Predicting the probabilities.

prediction_bow = Log_Reg.predict_proba(x_valid_bow)



prediction_bow

OUTPUT :-

Predicting the probabilities for a tweet falling into either Positive or Negative class.

If you are confused about the above output , read this stack overflow answer and you will have a clear idea about it.

The output basically provides us with the probabilities of the tweet falling into either of the classes that is Negative or Positive.

Calculating the F1 score

F1 Score

TF-IDF Features

Fitting the Logistic Regression Model.

Log_Reg.fit(x_train_tfidf,y_train_tfidf)

Predicting the probabilities.

prediction_tfidf = Log_Reg.predict_proba(x_valid_tfidf)



prediction_tfidf

OUTPUT :-

Predicting the probabilities for a tweet falling into either Positive or Negative class

Calculating the F1 Score

F1 Score

Note : In the nested list the element[0][0] is for label : 0 or Positive tweets and element [0][1] is for label : 1 or Negative Tweets.

For an in depth analysis of Logistic Regression do read the following article.

XGBoost

The next model we use is XGBoost.

from xgboost import XGBClassifier

Bag-of-Words Features

model_bow = XGBClassifier(random_state=22,learning_rate=0.9)

Fitting the XGBoost Model

model_bow.fit(x_train_bow, y_train_bow)

Predicting the probabilities.

xgb = model_bow.predict_proba(x_valid_bow)



xgb

Predicting the probability of a tweet falling into either Positive or Negative class.

Calculating the F1 Score

OUTPUT :-

F1 Score

TF-IDF Features

model_tfidf = XGBClassifier(random_state=29,learning_rate=0.7)

Fitting the XGBoost model

model_tfidf.fit(x_train_tfidf, y_train_tfidf)

Predicting the probabilities.

xgb_tfidf=model_tfidf.predict_proba(x_valid_tfidf)



xgb_tfidf

Predicting the probability of a tweet falling into either Positive or Negative class.

Calculating the F1 Score

OUTPUT :-

F1 Score

For a more in depth analysis of the XGBoost model read the following article.

Decision Trees

The last model we use is Decision Trees.

from sklearn.tree import DecisionTreeClassifier dct = DecisionTreeClassifier(criterion='entropy', random_state=1)

Bag-of-Words Features

Fitting the Decision Tree model.

dct.fit(x_train_bow,y_train_bow)

Predicting the probabilities.

dct_bow = dct.predict_proba(x_valid_bow)



dct_bow

Predicting the probability of a tweet falling into either Positive or Negative class

Calculating the F1 Score

F1 Score

TF-IDF Features

Fitting the Decision Tree model

dct.fit(x_train_tfidf,y_train_tfidf)

Predicting the probabilities.

dct_tfidf = dct.predict_proba(x_valid_tfidf)



dct_tfidf

Predicting the probability of a tweet falling into either Positive or Negative class

Calculating the F1 Score

F1 Score

For a more in depth analysis of Decision Trees model do give a read to the following article.

Model Comparison

Now, let us compare the different models we have applied on our dataset with different word embedding techniques.

Bag-of-Words

F1 Score of different models using features from Bag-of-Words

Comparison Graph