Social Media Sentiment Analysis using Machine Learning : Part — II | by Deepak Das | Towards Data Science

Hello everyone, so let’s start right where we left off in Part — I.
In this post we will discuss how we can extract features from our textual dataset by using Bag-of-Words and TF-IDF. Then we will see how we can apply Machine Learning models using these features to predict whether a tweet falls into the Positive: ‘0’ or Negative: ‘1’ sentiment.
Note : In case you haven’t read the Part — I of this series do give it a read to get a better understanding of Part — II.
medium.com
So, let’s start shall we ?
Bag of Words is a method to extract features from text documents. These features can be used for training machine learning algorithms. It creates a vocabulary of all the unique words occurring in all the documents in the training set.
Consider a corpus (a collection of texts) called C of D documents {d1,d2.....dD} and N unique tokens extracted out of the corpus C. The N tokens (words) will form a list, and the size of the bag-of-words matrix M will be given by D X N. Each row in the matrix M contains the frequency of tokens in document D(i).
For example, if you have 2 documents-
D1: He is a lazy boy. She is also lazy.
D2: Smith is a lazy person.
First, it creates a vocabulary using unique words from all the documents.
[‘He’ , ’She’ , ’lazy’ , ‘boy’ , ‘Smith’ , ’person’]
As we can see in the above list we don’t consider “is” , “a” , “also” in this set because they don’t convey the necessary information required for the model.
Here, D=2, N=6
The matrix M of size 2 X 6 will be represented as:
The above table depicts the training features containing term frequencies of each word in each document. This is called bag-of-words approach since the number of occurrence and not sequence or order of words matters in this approach.
So,let’s apply this word embedding technique to our available dataset.
We have a package called CountVectorizer to perform this task.
OUTPUT :-
TF-IDF stands for Term Frequency-Inverse Document Frequency, and the TF-IDF weight is a weight often used in information retrieval and text mining. This weight is a statistical measure used to evaluate how important a word is to a document in a collection or corpus. The importance increases proportionally to the number of times a word appears in the document but is offset by the frequency of the word in the corpus.
Typically, the TF-IDF weight is composed by two terms:
The first computes the normalized Term Frequency (TF), aka. the number of times a word appears in a document, divided by the total number of words in that document.
Example :-
Consider a document containing 100 words wherein the word cat appears 3 times.
The Term Frequency (TF) for cat is then (3 / 100) = 0.03
The second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.
Example :-
Assume we have 10 million documents and the word cat appears in one thousand of these.
Then, the Inverse Document Frequency (IDF) is calculated as
log(10,000,000 / 1,000) = 4.
Formula for finding the TF-IDF weight :-
From the above examples the Term Frequency is 0.03 and Inverse Document Frequency is 4.
Thus, the TF-IDF weight is the product of these quantities : 0.03 * 4 = 0.12.
CODE :-
Let us apply this technique to our dataset using Python.
We have a package available for this in Scikit-Learn known as TfidfVectorizer.
OUTPUT :-
These are the Word Embedding techniques which we have used on our dataset for feature extraction.
Let’s move on to the next step.
Splitting of our dataset into training and validation set.
From the above two techniques that is Bag-of-Words and TF-IDF we have extracted features from the tweets present in our dataset.
Now, we have one dataset with features from the Bag-of-Words model and another dataset with features from TF-IDF model.
First task is to split the dataset into training and validation set so that we can train and test our model before applying it to predict for unseen and unlabeled test data.
Using the features from Bag-of-Words for training set
train_bow = bow[:31962]train_bow.todense()
OUTPUT :-
Using features from TF-IDF for training set
train_tfidf_matrix = tfidf_matrix[:31962]train_tfidf_matrix.todense()
OUTPUT :-
Splitting the data into training and validation set
from sklearn.model_selection import train_test_split
Bag-of-Words Features
x_train_bow, x_valid_bow, y_train_bow, y_valid_bow = train_test_split(train_bow,train['label'],test_size=0.3,random_state=2)
TF-IDF features
x_train_tfidf, x_valid_tfidf, y_train_tfidf, y_valid_tfidf = train_test_split(train_tfidf_matrix,train['label'],test_size=0.3,random_state=17)
We are done with splitting our dataset into train and validation set.
Want to know more about the Bag-of-Words and TF-IDF models used for feature extraction. Do give a read to the following blog post for an in depth discussion.
towardsdatascience.com
Finally , we are here for the most awaited part of this series that is applying Machine Learning Models on our dataset.
The underlying problem we are going to solve comes under the Supervised Machine Learning category. So, let us have a brief discussion about this topic before moving on to apply different Machine Learning models on our dataset.
The majority of practical machine learning uses supervised learning.
Supervised learning is where you have input variables (x) and an output variable (Y) and you use an algorithm to learn the mapping function from the input to the output.
The goal is to approximate the mapping function so well that when you have new input data (x) that you can predict the output variables (Y) for that data.
It is called supervised learning because the process of an algorithm learning from the training dataset can be thought of as a teacher supervising the learning process. We know the correct answers, the algorithm iteratively makes predictions on the training data and is corrected by the teacher. Learning stops when the algorithm achieves an acceptable level of performance.
Supervised learning problems can be further grouped into regression and classification problems.
Classification: A classification problem is when the output variable is a category, such as “red” or “blue” or “disease” and “no disease” or in our case “Positive” or “Negative”
Regression: A regression problem is when the output variable is a real value, such as “dollars” or “weight”.
Our problem comes under the classification category because we have to classify our results into either Positive or Negative class.
There is another category of Machine Learning algorithm called Unsupervised Machine Learning where you have an input data but no corresponding output variables. The goal for unsupervised learning is to model the underlying structure or distribution in the data in order to learn more about the data. But that is of no concern for us for this problem statement.
So from the above splitting of dataset we see that we will use features from the Bag-of-Words and TF-IDF for our Machine Learning Models.
We generally use different models to see which best fits our dataset and then we use that model for predicting results on the test data.
Here we will use 3 different models
Logistic Regression
XGBoost
Decision Trees
and then we will compare their performance and choose the best possible model with the best possible feature extraction technique for predicting results on our test data.
We will use F1 Score throughout to asses our model’s performance instead of accuracy. You will get to know why at the end of this article.
CODE :-
from sklearn.metrics import f1_score
Now, let’s move on to applying different models on our dataset from the features extracted by using Bag-of-Words and TF-IDF.
The first model we are going to use is Logistic Regression.
from sklearn.linear_model import LogisticRegressionLog_Reg = LogisticRegression(random_state=0,solver='lbfgs')
Fitting the Logistic Regression Model.
Log_Reg.fit(x_train_bow,y_train_bow)
Predicting the probabilities.
prediction_bow = Log_Reg.predict_proba(x_valid_bow)prediction_bow
OUTPUT :-
If you are confused about the above output , read this stack overflow answer and you will have a clear idea about it.
stackoverflow.com
The output basically provides us with the probabilities of the tweet falling into either of the classes that is Negative or Positive.
Calculating the F1 score
Fitting the Logistic Regression Model.
Log_Reg.fit(x_train_tfidf,y_train_tfidf)
Predicting the probabilities.
prediction_tfidf = Log_Reg.predict_proba(x_valid_tfidf)prediction_tfidf
OUTPUT :-
Calculating the F1 Score
Note : In the nested list the element[0][0] is for label : 0 or Positive tweets and element [0][1] is for label : 1 or Negative Tweets.
For an in depth analysis of Logistic Regression do read the following article.
machinelearningmastery.com
The next model we use is XGBoost.
from xgboost import XGBClassifier
model_bow = XGBClassifier(random_state=22,learning_rate=0.9)
Fitting the XGBoost Model
model_bow.fit(x_train_bow, y_train_bow)
Predicting the probabilities.
xgb = model_bow.predict_proba(x_valid_bow)xgb
Calculating the F1 Score
OUTPUT :-
model_tfidf = XGBClassifier(random_state=29,learning_rate=0.7)
Fitting the XGBoost model
model_tfidf.fit(x_train_tfidf, y_train_tfidf)
Predicting the probabilities.
xgb_tfidf=model_tfidf.predict_proba(x_valid_tfidf)xgb_tfidf
Calculating the F1 Score
OUTPUT :-
For a more in depth analysis of the XGBoost model read the following article.
machinelearningmastery.com
The last model we use is Decision Trees.
from sklearn.tree import DecisionTreeClassifierdct = DecisionTreeClassifier(criterion='entropy', random_state=1)
Fitting the Decision Tree model.
dct.fit(x_train_bow,y_train_bow)
Predicting the probabilities.
dct_bow = dct.predict_proba(x_valid_bow)dct_bow
Calculating the F1 Score
Fitting the Decision Tree model
dct.fit(x_train_tfidf,y_train_tfidf)
Predicting the probabilities.
dct_tfidf = dct.predict_proba(x_valid_tfidf)dct_tfidf
Calculating the F1 Score
For a more in depth analysis of Decision Trees model do give a read to the following article.
towardsdatascience.com
Now, let us compare the different models we have applied on our dataset with different word embedding techniques.
Comparison Graph
Comparison Graph
As we can see the best possible model from both Bag-of-Words and TF-IDF is Logistic Regression.
Now, let us compare the score of the Logistic Regression model with both the feature extraction techniques that is Bag-of-Words and TF-IDF.
Comparison Graph
From the above comparison graph we can clearly see that the best possible F1 Score is obtained by the Logistic Regression Model using TF-IDF features.
Code :-
res = pd.read_csv('result.csv')res
From the above output we can see that our Logistic Regression model with TF-IDF features predicts whether a tweets falls into the category of Positive — label : 0 or Negative — label : 1 sentiment.
Bag-of-Words
TF-IDF
Logistic Regression
XGBoost
Decision Trees
F1 Score
So, finally we have reached the end of our journey. We completed the tasks which were required to predict the sentiment of a particular tweet using Machine Learning.
The questions that arises are “What is F1 Score ?” and “Why F1 Score instead of accuracy ?”.
So, before we proceed you need to have a basic idea about the terminologies like Confusion Matrix and its contents for example.
So, refer to this article for a basic understanding of the terminologies associated with Confusion Matrix.
towardsdatascience.com
OK , let us answer the above queries.
Let us generate a countplot for our training dataset labels i.e. ‘0’ or ‘1’ .
sns.countplot(train_original['label'])sns.despine()
From the above countplot generated above we see how imbalanced our dataset is.We can see that the values with Positive — label : 0 sentiments are quite high in number as compared to the values with Negative — label : 1 sentiments.
So when we keep Accuracy as our evaluation metric there may be cases where we may encounter high number of false positives. So that is why we use F1 Score as our evaluation metric instead of Accuracy.
To know about F1 Score we first have to know about Precision and Recall.
Precision means the percentage of your results which are relevant.
Recall refers to the percentage of total relevant results correctly classified by your algorithm.
We always face a trade-off situation between Precision and Recall i.e. High Precision gives low Recall and vice versa.
In most problems, you could either give a higher priority to maximising precision, or recall, depending upon the problem you are trying to solve. But in general, there is a simpler metric which takes into account both precision and recall, and therefore, you can aim to maximise this number to make your model better. This metric is known as F1-score, which is simply the harmonic mean of Precision and Recall.
So this metric seems much more easier and convenient to work with, as you only have to maximise one score, rather than balancing two separate scores.
Finally, we have reached the end of the two part series of this article. I hope after this post you get a basic understanding of how you can start off with text processing and apply Machine Learning models to text data and extract information out of it.
After this we can try to deploy our Machine Learning Models over a website using the available web frameworks such as Flask ,FastAPI etc. to production. But that’s a story for another blog post.
You can reach me at :
LinkedIn : https://www.linkedin.com/in/deepak-das-profile/
GitHub : https://github.com/dD2405