10 Tasks With NLP: Twitter Sentiment Analysis
By: Kurt F.
Tags: NLP, Twitter, Data Analysis, Data Science, Sentiment Analysis

Task-1: Understanding the Business Problem

As always, we address our business problem in the first step of the project. In this study, we hope to get a result with two positive and negative results. We will process user comments in a sample data set obtained through the Twit API. With NLP techniques, we aim to obtain a numerical representation and enable the expression to clearly describe a real emotion.

Task-2: Import Libraries and Dataset

We usually start by installing libraries that are frequently used in data analysis. Those:

pandas (import pandas as pd)

(import pandas as pd) numpy (import numpy as np)

(import numpy as np) seaborn (import seaborn as sns)

(import seaborn as sns) matplotlib.pyplot (import matplotlib.pyplot as plt)

Also, a little touch of my notebook is required for places where I think we can express it better in black. To set the Jupyter notebook theme, we do as follows:

jupyterthemes

For installation: pip install jupyterthemes

For import: from jupyterthemes import jtplot

Theme of our choice: jtplot.style(theme=’monokai’, context=’notebook’, ticks=True, grid=False)

Note: If you want to see the codes I mentioned here clearly in the notebook in the link, I suggest you make the theme setting. However, this is completely optional.

The data set / data source we use:

32K tweets labeled as hate (racist / sexist) or non-hate in this link:

After loading the dataset as below, you will realize that we have to make some decisions about the columns. Because if we do not eliminate the details that we decided to be useless in an analysis, we will both establish a cumbersome system and reduce the effect of our analysis.

That’s why we decided to remove the “id” column as follows:

Task-3: Explore Dataset

At this stage, we will apply some methods to discover our data set.

1.Method: Are there any empty values in the data set? To do this, let’s try to combine the isnull() query with a graphical representation using the seaborn library.

As you can see, no blank values are displayed.

2. Method: Let’s draw a histogram for a graph of data distribution. The histogram graph will be very useful if the data frame is of a similar scale.

3. Method: Let’s find out the length of the twitts

4. Method: Let’s visualize this with matplotlib

5. Method: Let’s define the tweets. For this: describe()

6. Method: Now let’s see what is the shortest and longest message. As we understand from the above table, min = 11, max = 84 then for the shortest message:

Longest message:

7.Method: Finally, let’s examine how many data we have labeled as negative or positive. We’re looking at those labeled with 1 for negative, 0 for positive. So, positive:

For negative:

Task-4: Plot the WordCloud

Let’s try to combine and count sentences with tolist (), which we can use in Pandas series.

And let’s put it all together like a sentence. Thus, we will be able to use them more comfortably for WordCloud.

Let’s do the necessary downloads for WordCloud as follows:

!pip install WordCloud

from wordcloud import WordCloud plt.figure(figsize=(20,20))

plt.imshow(WordCloud().generate(sentences_as_one_string))

Let’s create a sample WordCloud for negative statements:

Task-5: Perform Data Cleaning

At this stage we will learn how to clear our data. What we mean is a stage that should be well known in a classic NLP project. Actually, everything up to this point has been the provision of prerequisites for starting the NLP project. Now let’s clean up expressions with Punctuation and Stopwords.

1- Remove Punctuation from Text: The string is used to work with classes and constants related to text. Punctuation helps to remove unwanted punctuation marks in text. In order to use the punctuation function, we first need to import string as below.

Unwanted punctuation marks are output

As an example, let’s try to understand how it works with a sentence. Sample sentence: “Good morning beautiful people :)… I am having fun learning Machine learning and AI!!”

The expression flows down, but for a better representation, we may prefer:

So we could see more clearly how the punctuation function works.

Alternatively, we could do this with a for loop like the following:

2- Remove ‘Stopwords’: Stopwords performs the function of extracting words such as “the”, “and”, “is” in the text. Thus, vectorization of more specific words for analysis is provided. For this you have to download the Natural Language tool kit and the Stopwords package to execute this command as follows:

Let’s take a closer look at the use of StopWords based on the sentence we have given above:

Test_punct_removed_join_clean = [word for word in Test_punct_removed_join.split() if word.lower() not in stopwords.words('english')]

Note: ‘StopWords’ separates words they think are necessary. However, it does not interfere with the basic structure of the sentence regarding its meaning. As we mentioned earlier, the important thing is to focus on the words that convey the emotion.

Task-6: Perform Count Vectorization (Tokenization)

An NLP project cannot be completed without tokenization. Tokenization can be based on text in three different ways. These are: word-character and subword (n-gram character). For example: “this word”:

Word _tokenized→this-is-word

Character_tokenized →t-h-i-s-i-s-w-o-r-d

For a better understanding of the subword, see the image below:

image created by the author

Our task here is to display the data in a vector format in addition to all this preliminary information. For this, to install sklearn, and also CountVectorizer must be imported from sklearn.feature_extraction.text.

!pip install sklearn

from sklearn.feature_extraction.text import CountVectorizer

For the vectorization function: CountVectorizer()

Data fit and transformation: fit_transform()

To print which words to show as ‘vectorizable’: get_feature_names()

Now let’s make an example variable with a data_set to better understand the above:

Task-7: Create a Pipline to Remove Punctuations, Stopwords and Perform Count Vectorization

Now let’s go back to our dataset and see how we can create the above together.

Let’s define a pipeline to clean up all the messages. The pipeline performs the following: (1) Remove punctuation, (2) Remove stopwords

The function we will define should be as follows:

def message_cleaning(message):

Test_punc_removed = [char for char in message if char not in string.punctuation]

Test_punc_removed_join = ''.join(Test_punc_removed)

Test_punc_removed_join_clean = [word for word in Test_punc_removed_join.split() if word.lower() not in stopwords.words('english')]

return Test_punc_removed_join_clean

If we test this functionality for any tweet:

Now, define the cleaning pipeline we defined earlier.

from sklearn.feature_extraction.text import CountVectorizer



vectorizer = CountVectorizer(analyzer = message_cleaning)

tweets_countvectorizer = CountVectorizer(analyzer = message_cleaning, dtype = 'uint8').fit_transform(tweets_df['tweet']).toarray()

And let’s see what the qualified words are and examine as vectorial.

print(vectorizer.get_feature_names())

print(tweets_countvectorizer.toarray())

You can get the number of rows of columns with the shape () function.

Note: As you can see from the numbers here, displaying this data as an array may cause problems on some machines. For this, I am leaving a link that I find useful about the ‘MemoryError: Unable to allocate…’ error in stockoverflow. If you get uncorfortable, you can look for a solution.

Now, before starting the test, we can assign variables as x and y as in the classic two-variable. For this, we can create equations for tweets_count_vectorizer i to variable x and to variable y for label. Thus, they are ready for training to easily create our machine learning model.

X =tweets_countvectorizer

y =y =tweets_df['label']

Task-8: Understand the Theory and Intuition Behind NAIVE BAYES

Naive Bayes is a classification technique based Bayes Theorem. The Naive Bayes Classifier assumes that the presence of a particular feature in a class has nothing to do with the existence of any other property. The level of relationship is determined between variables in a similar cluster, such as those who are members of a campaign according to customer demographic structure and population distribution, but separated by different preferences. Probability calculations are made accordingly. In principle, it works step by step as follows:

Calculating the first probability for the given class labels: This is the probability calculation within the total of the individual groups according to the available data. For example, calculating the odds of 4 red or 6 blue balls between 10 balls, so the probability is calculated as 4/10 for a red ball and 6/10 for blue.

This is the probability calculation within the total of the individual groups according to the available data. For example, calculating the odds of 4 red or 6 blue balls between 10 balls, so the probability is calculated as 4/10 for a red ball and 6/10 for blue. Finding the likelihood with each attribute for each class: We calculate the probability of the new variable at a location between two separate variables by looking at its specific location. For example, look at how many blue and red neighbors are next to them between the blue and red balls as indicated by the numbers above. If there are 3 red and 1 blue balls around the object (in the nearest circle) for which we are investigating the probability of the number, it is calculated as 3/4 and 1/6.

We calculate the probability of the new variable at a location between two separate variables by looking at its specific location. For example, look at how many blue and red neighbors are next to them between the blue and red balls as indicated by the numbers above. If there are 3 red and 1 blue balls around the object (in the nearest circle) for which we are investigating the probability of the number, it is calculated as 3/4 and 1/6. Posterior Probability: The above two calculations are set for each variable. Accordingly, for red: 4/10 * 3/4 = 3/10 and for blue: 6/10 * 1/6 = 1/10.

The above two calculations are set for each variable. Accordingly, for red: 4/10 * 3/4 = 3/10 and for blue: 6/10 * 1/6 = 1/10. Solving these values according to the Bayes Formula and calculating the final probability:

Task-9: Train a NAIVE BAYES Classifier Model

So now we have a tokenized version (X) of all the tweets. We previously specified it as X = tweets_countvectorizer.

Let’s import our function from the sklearn library to split arrays or matrices into random subdirectories:

the data allocated for testing the shift

We want to use the Multinomial Naive Bayes classifier. Because it is suitable for text classification with word numbers. We will use multinomialNB () by assigning it to a variable. It could feed again. Feeding in the data to the input and output basically is here. We are training the model on 32K tweets. Therefore, this process may take some time. In that case:

the model is trained, everytihng looks good

Task-10: Assess Trained Model Performance

We will finalize our project by evaluating the model performance. First, let’s try to understand confusion matrixes. Which is simply a way visualizing the performance of our classifier. With Sklearn, the confusion matrix function is used to evaluate and calculate the accuracy of our classification.

In fact, the negative measurement of a message estimation model that is actually negative will be output without error. We can get two types of errors. These:

type 1 Error: If the model responds positively to a message that is actually negative

If the model responds positively to a message that is actually negative type 2 Error: If the model responds negatively to a message that is actually positive

NB_classifier was our train model. We will generate y_predict_test by feeding it with X test. In the confusion matrix we will try to visualize the results with a heatmap. Innovations will be equals to true simply

It has missclassified these two samples missclassified 1.7 and 2.1. However, he was able to classify around 5800 tweets as positive and around 2500 as negative. (We used 20% of the total data as test data, as we mentioned above [size = 0.2])

So let’s go ahead and print out classification report.

Not bad with an overall weighted average of 0.94. However, we should not forget that our dataset is an unbalanced dataset. f1score which is the harmonic mean between the ‘precision’ and ‘recall’ tells us that the model can be improved.