Sentiment Analysis on Swachh Bharat using Twitter
By: Siva Sai
Tags: Machine Learning, Sentiment Analysis, Twitter, Naturallanguageprocessing, Data Science

Sentiment analysis is extracting the perception of people towards a particular issue, brand, scheme, etc., (sentiment) from textual data. It has a wide range of applications from brand-monitoring, product-review analysis to policy framing. In this article, we perform Sentiment analysis on tweets with Swachh Bharat hashtag. Full code of this project is available here.

This article is divided into the following parts.

Different approaches for performing Sentiment Analysis Collecting Data Converting useful data to CSV Dropping duplicates rows and unnecessary columns Cleaning the tweets Stop Words Removal Lemmatizing and Stemming Data Visualisation Vectorizing the text of tweets Implementing classifiers from sklearn Implementing Gensim models Summary

Different approaches for performing Sentiment Analysis

There are two main approaches for performing Sentiment analysis-lexicon based approach or using Machine Learning models. Any Lexicon based approach fundamentally uses a dictionary of words with semantic scores assigned to them to calculate a final polarity of a tweet. For example, consider the sentence “I love this movie, even though the second part is a bit slow”.Now a (total)sentiment score will be assigned to each of the words like this.

“I:0, love:0.750, this:0.0, movie:0.0, eventhough:0.0 the:0.0 second :0.0,part:0.0,is :0.0,boring:-0.250”

A positive score denotes that the word is positive and a negative score implies it is negative. Now, if we add up all the scores for this sentence, it will be-0.500, which is a positive sentiment. So the tweet will be labeled as positive. Few imporatant lexicon resources available are- SentiWordNet, TextBlob, Afinn, The MPQA (Multi-Perspective Question Answering) Subjectivity Lexicon and Pattern. I used this lexicon approach using SentiWordNet to label my tweets(explained later).

The second, machine learning approach, tackles the problem as a text classification task employing classifiers after pre-processing the data. We follow the latter approach, which will be explained in detail.

Regarding the tools required for this task

There are many libraries for text processing and classification-NLTK(comprehensive and a hard learning curve), TextBlob(Easy to use , suitable for beginners), Spacy(ideal for industry-work), and many more. I choose to go for NLTK because it gives many support libraries to implement many text-processing techniques

Let’s start!

Collecting Data

Now I need textual data which expresses people’s sentiment about Swachh Bharat. Few options available are using Reddit API, Facebook API, Twitter API, and scraping data from web pages. I decided to go with twitter API to collect tweets regarding Swachh Bharat, as many consider Twitter as a big psychological database. The tweets for this project are collected from the Twitter database using Tweepy library in python. In order to access the tweets, one has to create a twitter-developer account. There are different types of developer accounts: Standard Search(free), Premium(paid) and Enterprise(paid). The features of the API or the accessibility increases from Standard account to an Enterprise account. For example, through Standard account, you can access tweets of the last seven days while this is 30 incase of Premium. For creating the account, you have to provide the details of the core use case, intent, business purpose of your use , details about the analyses you plan to conduct, and the methods or techniques. And I advise to provide as more details as much as possible(can also add webpage links to the project similar to yours or the main topic of the project whenever you want to clarify things) in the first attempt itself, else you have to write long emails re-explaining the same content again and again in case they need more information. The Twitter team will thoroughly review your application. This is a good article about Twitter Data Mining.

It usually takes less than a day to get an approval mail, provided that your information is complete.

After this, you need to create a new app in their interface and provide its description(as below) which generates API KEY, API SECRET KEY, ACCESS TOKEN, and ACCESS TOKEN SECRET, which we will use to log in through terminal.

Creating a new application

Now let’s authenticate with our details using tweepy.AppAuthHandler() function. Replace ‘*’s with your authentication details.

import tweepy

API_KEY="*********************"

API_SECRET="*************************"

ACCESS_TOKEN="*******************************"

ACCESS_TOKEN_SECRET="******************************"

auth = tweepy.AppAuthHandler(API_KEY, API_SECRET)

auth.set_access_token(ACCESS_TOKEN, ACCESS_TOKEN_SECRET)

api = tweepy.API(auth)

print(api.me().name)

If the authentication was successful, you should see the name of your account printed.

IAmSai

After authentication, we are all set to download the tweets. We use API.search() method to fetch tweets. There are two main ways of searching for tweets with Tweepy-search by a hashtag(or a keyword) and search by the id of a tweet( id of each tweet is unique). The second method is useful when you have all the id’s of tweets to be collected(Say, you want to collect tweets about Demonetisation given the id’s.These id’s will be given by those who has previously collected data about that topic). Since I don’t have any id’s I go with the first method.

search_query="#SwachhBharat"

new_tweets = api.search(q=search_query,count=tweetsPerQuery,lang="en",tweet_mode='extended')

auth = tweepy.AppAuthHandler(API_KEY,API_SECRET)

api = tweepy.API(auth,wait_on_rate_limit=True,wait_on_rate_limit_notify=True)

fName = 'sb_04_08_19.txt' # where i save the tweets

Few important lines of code are given in the above code snippet. Please see this for detailed code.

We search using the keyword “#SwachhBharat”. Later, while cleaning data, I realized that tweets with hashtags-’#MyCleanIndia’ and ‘#SwachhBharatUrban’ can also be used to search for tweets regarding Swachh Bharat. We use AppAuthHandler(3rd line)rather than mostly used-OAuthHandler for authorizing the API.OAuthHandler allows a maximum of 180 requests per 15-minute window(and a maximum of 100 tweets per request). So we can fetch only 18,000 tweets in 15 minutes. But AppAuthHandler has its maximum request rate limit as 450 and 100 tweets per request. Thus it helps in fetching 45,000 tweets in fifteen minutes. Also, I used extended-tweet mode(second line) to make sure that we get full tweets rather than truncated ones. After downloading, we will save the tweets in a text file(alternatively, we can also store them in a .json file as well). I have repeated the above procedure a few more times to collect more number of tweets later(with a gap of more than seven days between two data collections, else we’ll get a lot of duplicate tweets).

Downloading the tweets

Converting useful data to CSV

This was the difficult part for me to figure out initially, as I have never worked with APIs and JSON. The text file is a huge one with around 1100 pages of information. In ‘sb.txt’(text file) , the tweets are in JSON (JavaScriptObjectNotation) format .Take a look at it.

Data is full of many JSON objects

Just by a glance, you can see that there are many features(objects) which are unimportant to our task like- metadata, indices, screen_name. So I went through this page to figure out the objects I may need for my task. I decided to go for these objects- full_text(full text of the tweet), retweet_count(of a tweet), favorite_count(of a tweet), geo(geographic location), coordinates(latitude and longitude coordinates of the place ), place(from which it is tweeted), follower_count(of the user who tweeted), created_at(time and date of tweet), id_str(id of the tweet in a string format). I keep the above objects in my CSV file and discard the rest.

inputFile='sb_04_08_19.txt'

tweets = []

for line in open(inputFile, 'r'):

tweets.append(json.loads(line))

for tweet in tweets:

try:

csvWriter.writerow([tweet['full_text'],tweet['retweet_count'],tweet['user']['followers_count'],tweet['favorite_count'],tweet['place'],tweet['coordinates'],tweet['geo'],tweet['created_at'],str(tweet['id_str'])])

count_lines+=1

except Exception as e:

print(e)

Few important lines of code to convert from the text file to CSV are given above. Please see this for more clarity. We create a list of tweets reading each line from the file(1st for-loop). Then we use csvWriter.writerow() to write the necessary information to a CSV file for each tweet. And we only collect the objects mentioned above.

Let us try to load the CSV-file in Jupyter Notebook. It’s essential to set the encoding parameter as ‘unicode_escape’ in read_csv() function as some tweets also consist of emojis and would make it difficult with default encoding.

import pandas as pd

df = pd.read_csv('somall.csv', encoding = 'unicode_escape')

df.head(5)

Have a look at the output.

Dropping duplicates rows

Let us see and delete duplicate rows if any. There are two types of duplicates-duplicates with same values for all columns(this duplication happens when same tweets are collected again by tweet-collector) and duplicates with the same text for tweets(This occurs when two or more users post the same tweet.)

print(len(df.index))#14195

serlis=df.duplicated().tolist()

print(serlis.count(True))#112

serlis=df.duplicated(['full_text']).tolist()

print(serlis.count(True))#8585

We can see that there are 112 duplicate rows and 8585 rows which are duplicated in the column-full_text and row-duplicates are a subset of full_text-duplicates. So we drop all the duplicate rows.

df=df.drop_duplicates(['full_text'])

Initially, I didn’t notice the duplicates. So, when the text classifiers are used, they were giving me high accuracies as high as 99.54%, which are not good.

Dropping unnecessary columns

It seems a few features like ‘place’, ‘geo’ have more not-a-number(NaN) as their values. Let’s check how many empty rows exist in some selected columns

print(df['geo'].isna().sum())

print(df['place'].isna().sum())

print(df['coordinates'].isna().sum())

There are 14,195 rows totally in our dataset. We see that 14191 rows of geo-column are empty. Empty rows in’ place’ and ‘coordinates’ are also high(13951 and 14191). So we drop these columns along with ‘id_str’ (which is the id of tweet) as tweet id is not of much use for us. Although we are not using retweet_count, user_followers_count, favorite_count in our context, we are not dropping them because they can be used for other tasks(like estimating the popularity of tweet which can be used to assign weights to tweets).

df=df.drop([‘place’,’coordinates’,’geo’,’id_str’],axis=1)

df.head(5)

First five rows of the dataset after dropping a few columns

Cleaning the tweets

In the text of a tweet, there may be some unnecessary symbols which is not essential for our analysis. So let’s go ahead and clean our tweets.

Let’s observe the full text of any particular tweet.

full text of a tweet at row number 22

As we can see, some of the unnecessary text and symbols to be removed are — username_tags(like @_WeMeanToClean), retweet symbol(RT), hashtags(like #MyCleanIndia), URLs(like https://t.co/XK3ZReâ\x80 ),numbers and punctuations .Some of the meaningful hashtags convey meaning and can have some sentiment in it after the word is segmented into useful parts (like #ILoveSwachhBharat). So, instead of removing all the words starting with hashtag symbols, only ‘#’ symbols are removed. We perform this text cleaning using re module in python. The re.sub() function searches for a pattern and replaces with the text we specify. We replace all these symbols with a whitespace character.

import re

for i in range(len(df)):

txt = df.loc[i]["full_text"]

txt=re.sub(r'@[A-Z0-9a-z_:]+','',txt)#replace username-tags

txt=re.sub(r'^[RT]+','',txt)#replace RT-tags

txt = re.sub('https?://[A-Za-z0-9./]+','',txt)#replace URLs

txt=re.sub("[^a-zA-Z]", " ",txt)#replace hashtags

df.at[i,"full_text"]=txt

Now, we can see that our tweets appear clean.

A tweet after cleaning

POS-Tagging and Sentiment labeling

We are done with the basic cleaning part of text data. In the ML algorithms that we are going to implement ‘full_text’ of the tweet acts as a predictor variable(other variables that can be used are retweet-count, favorite-count if we want to predict the impact of a tweet, but that’s not a part of our task currently). As it is evident, we need to create target variables (sentiment scores) for our data. For this purpose, we use SentiWordNet. SentiWordNet is an enhanced lexical resource explicitly devised for supporting sentiment classification and opinion mining applications. It has a large corpus of POS-tagged English words along with their sentiment.

Predicting sentiment of few words using SentiWordNet-An example

SWN gives pos_score and neg_score for each word as in the above case. The higher the pos_score, the more positive is the word. We use a simple linear summation of these scores(We add pos_score of all the words in a tweet to form a pos_total and in a similar way, we obtain neg_total. Then we add these two to obtain sent_total) and label a sentence as positive(1) if it(sent_total) is greater than 0, negative(-1) if it is less than 0 and neutral(0), otherwise. As you may have figured out, we need to find POS-tags(Parts of Speech-tags) of our tweets in order to use SWN. The pos_tag feature of nltk helps us to tag the text, but it is not simplified(Besides noun, pronoun, adjective, verb it also tags text with many more tags like adposition, conjunction, determiner, etc.,). But SWN accepts only simplified tags. So we try to simplify the tags using map_tag function from nltk library.

for i in range(len(df_copy.index)):

text = df_copy.loc[i]['full_text']

tokens = nltk.word_tokenize(text)

tagged_sent = pos_tag(tokens)

store_it = [(word, map_tag('en-ptb', 'universal', tag)) for word, tag in tagged_sent]

The second line converts the tweet text into tokens which will be tagged later with their respective Parts-of-Speech. Then we store the words and tags in store_it list. This list will subsequently be passed for sentiment labeling using SWN.

The tags of a tweet(“A door to door waste segregation awareness drive was carried out by members of nehru yuva kendra in kapurthala punjab”) when tagged using pos_tag() and simplified POS-tag are presented below.

Tagged Parts of Speech: [('a', 'DT'), ('door', 'NN'), ('to', 'TO'), ('door', 'VB'), ('waste', 'NN'), ('segregation', 'NN'), ('awareness', 'NN'), ('drive', 'NN'), ('was', 'VBD'), ('carried', 'VBN'), ('out', 'RP'), ('by', 'IN'), ('members', 'NNS'), ('of', 'IN'), ('nehru', 'JJ'), ('yuva', 'NN'), ('kendra', 'NN'), ('in', 'IN'), ('kapurthala', 'NN'), ('punjab', 'NN')] Tagged Parts of Speech- Simplified: [('a', 'DET'), ('door', 'NOUN'), ('to', 'PRT'), ('door', 'VERB'), ('waste', 'NOUN'), ('segregation', 'NOUN'), ('awareness', 'NOUN'), ('drive', 'NOUN'), ('was', 'VERB'), ('carried', 'VERB'), ('out', 'PRT'), ('by', 'ADP'), ('members', 'NOUN'), ('of', 'ADP'), ('nehru', 'ADJ'), ('yuva', 'NOUN'), ('kendra', 'NOUN'), ('in', 'ADP'), ('kapurthala', 'NOUN'), ('punjab', 'NOUN')]

Then we find the sentiment of the tweet by the method, mentioned above in the introduction. Please see the complete code for more clarity on sentiment labeling. After obtaining sentiments from SWN, we add three more columns to our CSV- ’pos_score’, ’neg_score’, ’sent_score’. The ‘pos_score’ of a tweet is the net positive score of all words in a tweet and the similar goes with ‘neg_score’.

Dataset after adding sentiment score for tweets

A value of 1 is assigned for a positive tweet( pos_score>neg_score), 0 for a neutral tweet(pos_score=neg_score) and -1 for a negative tweet(pos_score<neg_score). There are 3233 positive tweets, 814 neutral tweets, and 1563 negative tweets. I also used TextBlob and Afinn for sentiment assigning to tweets. The results are as follows:

TEXTBLOB:

Total tweets with sentiment: 5610

positive tweets: 2171

negative tweets: 591

neutral tweets: 2848 AFINN:

Total tweets: 5610

positive tweets: 2551

negative tweets: 572

neutral tweets: 2487

The proportion of the number of tweets from each sentiment in each classifier is consistent, giving us more confidence about sentiment labeling. Also, we can deduce that the majority of the tweets are positive.

Now we are done with sentiment labeling, we’ll prepare our text data further to feed into ML classifiers.

Stop Words Removal

Stop words are commonly used words like ‘of’, ’the’, for’ etc., In our context, they don’t add much weight to the sentiment. Also, they increase the dimensionality of data. So we remove the stop words using nltk library(in next part).

Before feeding this text data to ML algorithms, we convert it to vectors(in later parts of the article). And we don’t want to have different (vector)values for similar words. We expect the same values for all these words — ‘distribute’, ‘distribution’, ‘distributed’, ‘distributing’ as they essentially convey the same meaning. This(assigning same values to similar words) is achieved in a better way if we perform Lemmatizing and Stemming using nltk module.

Lemmatizing and Stemming

Stemming reduces the words to their stems. Stemming algorithms are majorly rule-based ones. For instance, they reduce all the above-’distribute’ words to ‘distribut’. A lemmatizer does the same thing as Stemmer but keeping the linguistics of the word in context. For instance, all the above words will be condensed to ‘distribute’.

from nltk.corpus import stopwords

from nltk.stem import WordNetLemmatizer

from nltk.stem import PorterStemmer

pstem = PorterStemmer()

lem = WordNetLemmatizer()

stop_words = stopwords.words('english') for i in range(len(df.index)):

text = df.loc[i]['full_text']

tokens = nltk.word_tokenize(text)

tokens = [word for word in tokens if word not in stop_words] for j in range(len(tokens)):

tokens[j] = lem.lemmatize(tokens[j])

tokens[j] = pstem.stem(tokens[j]) tokens_sent=' '.join(tokens)

df.at[i,"full_text"] = tokens_sent

The third line removes the stop words in tokens-list. After lemmatizing and stemming, we finally substitute the new text in place of the old one.

It is worth noting that the above three cleaning processes(Stop words Removal, Stemming, Lemmatizing) are not performed before(sentiment labeling) because doing so will cause irregularities in sentiment detection. The words like ‘distribut’, ‘cleanin’ can’t be found in SWN’s corpus of English words, thus SWN fails to detect positive and negative score of these words. This makes tweets from only a part of the dataset labeled with a sentiment score. We also convert all tweets to lower case using nltk library.

Data Visualisation

We use WordCloud to represent the majority of words in each of the sentiment. For this, we add all the text of tweets which has a positive label to form a string and denote this string as pos_string. We follow the same method for negative and neutral text too.

for i in range(len(df_copy.index)):

if(df_copy.loc[i]["sent_score"]==1):

pos_text+=df_copy.loc[i]["full_text"]

elif(df_copy.loc[i]["sent_score"]==-1):

neg_text+=df_copy.loc[i]["full_text"]

else:

neut_text+=df_copy.loc[i]["full_text"]

The text representations are generated using the WordCloud module in python and plotted using matplotlib.

list_text = [pos_text,neg_text,neut_text]

for txt in list_text:

word_cloud = WordCloud(width = 600,height = 600,max_font_size = 200).generate(txt)

plt.figure(figsize=(12,10))# create a new figure

plt.imshow(word_cloud,interpolation="bilinear")

plt.axis("off")

plt.show()

We should remember that the words have been stemmed and lemmatized before, so we may not be able to see the full words of a few.

Positive sentiment WordCloud

As you can see, the hashtags consist of important words in positive-text Wordcloud. It is quite intuitive as we didn’t remove the hashtags from the text altogether. Although a Hindi word, ‘swachh’(a positive word) appears quite often in this. There are other useful words like waste segregation, garbage collection, clean, etc.,

Neutral sentiment WordCloud

In neutral WordCloud, excluding a few hashtags, we can see many neutral words like- part, street, road, surroundings, internship, etc.,

Negative sentiment WordCloud

Stop littering, drain back, drain are few negative words from the cloud and we can also see that it contain many words from other sentiments.

Splitting the data

We split our data into training, validation and test sets using train_test_split function of model_selection module from nltk library. The division is as follows — 90% of training data, 5% validation data, and 5% test data. More proportion of data is for training, keeping in mind the less number of tweets.

import sklearn

from sklearn.model_selection import train_test_split

SEED=4

x = df.full_text

y = df.sent_score

x_train,x_val_test,y_train,y_val_test = train_test_split(x,y,test_size=0.1,random_state=SEED)

x_val,x_test,y_val,y_test = train_test_split(x_val_test,y_val_test,test_size=0.5,random_state=SEED)

Vectorizing the text of tweets

Before we implement different ML text classifiers, we need to convert the text data into vectors. It is crucial as the algorithms expect data in some mathematical for rather than textual form. This Quora answer explains why the conversion to vector spaces is essential. Here, we implement two vectorizers from sklearn- Count Vectorizer and Tfidf Vectorizer. Both of them comes under Bag-of-Words models.

Count Vectorizer counts the number of times a word appears in the document(in each tweet) and uses this value as its weight. To visualize, the columns in the vector(after vectorizing) consists of all distinct words in all the tweets and each row contain the tweets. Counter Vectorizer gives us a sparse matrix(filled with many zeros) as the vector.

from sklearn.feature_extraction.text import CountVectorizer

cv=CountVectorizer(decode_error='ignore',lowercase=False,max_features=11)

x_traincv=cv.fit_transform(x_train.values.astype('U'))

Let’s see the top words(the words which have the maximum frequency in all tweets combined) in CountVectorizer. See this for code for visualizing top words.

Count Vectorizer-Top words

Let’s implement Tfidf Vectorizer as well. TF-IDF stands for “term frequency-inverse document frequency”. The weight assigned to a token is the product of the term(word) frequency and the inverse of document frequency of the word. Thus, if a word appears more often in the tweets, it is assigned less importance(IDF-weight) in this vectorizer.

from sklearn.feature_extraction.text import TfidfVectorizer

tf=TfidfVectorizer(decode_error='ignore',lowercase=False,max_features=11)

x_traintf=tf.fit_transform(x_train_copy.values.astype('U'))

And the top words for Tfidf are:

Top words- Tfidf Vectorizer

Comparing top words in CV and Tfidf, we see that although the words are the same, they differ in their order and frequency. Also, Tfidf has fractional values for frequency, whereas CV has only integer values.

N-grams

In simple words, n-grams are all combinations of adjacent words that we can find in our text(tweets). All combinations possible for the sentence “I like toys”: unigrams-(‘I’, ’like’, ’toys’), bigrams-(‘I like’, ‘like toys’), trigrams-(‘I like toys’). The word ‘Swachh’ is more often followed by ‘Bharat’. Thus bi-grams provides us ‘Swachh Bharat’ as an additional feature along with ‘Swachh’ and ‘Bharat’. Both CV and Tfidf gives a parameter called ngram_range. The ngram_range as (1,1) denotes uni-gram, (1,2) a bigram, (1,3) a trigram and so on. We implement uni, bi, and trigrams with both CV and Tfidf and observe which of them gives more accuracy score.

Implementing classifiers from sklearn

After vectorizing our tweets, we are all set to implement classification algorithms. The classifiers we are going to implement are-Naive Bayes models(MultinomialNB, BernoulliNB,), Linear models (LogisticRegression, RidgeClassifier, PassiveAggressiveClassifier, Perceptron) , Ensemble models( RandomForest classifier, AdaBoostClassifier) and SVM model(LinearSVC). We use the accuracy score to measure the performance of the model (precision score, recall and confusion matrix are also calculated).

K-fold cross-validation is performed with each classifier to verify the consistency in the performance of the model using cross_validate module in sklearn. The number of folds is set to 10 and the mean and standard deviation of cross-validation scores are noted. The time taken for training each classifier is also measured using time.time() function. After this, the results are stored in a CSV file. Few important(not all) lines of code are given below. Please see this for full code.

classifiers = [MultinomialNB(),BernoulliNB(),LogisticRegression(),LinearSVC(),AdaBoostClassifier(),RidgeClassifier(),PassiveAggressiveClassifier(),Perceptron(),RandomForestClassifier()]

for clf in classifiers:

model = make_pipeline(vec,clf)#vec is CV or Tfidf

model.fit(x_train.values.astype('U'),y_train.values.astype('U'))

labels = model.predict(x_val_copy.values.astype('U'))

ac = accuracy_score(y_val_copy.values.astype('U'),labels)

kfold = KFold(n_splits=10,shuffle=False,random_state=None)

results = cross_validate( model,x_train_copy.values.astype('U'), y_train_copy.values.astype('U'),cv=kfold)



data.append([vec_gram,clf_names[i],ac,crossval_train_score_mean,crossval_test_score_mean,crossval_train_score_std,crossval_test_score_std, end-start])

i+=1

d = pd.DataFrame(data,columns=['Vec_Gram','Classifier','Ac','crossval_train_score_mean','crossval_test_score_mean','crossval_train_score_std','crossval_test_score_std','Time.2'])

d.to_csv('all_clfs.csv')

The results are as follows.

Results of all classifiers using Count Vectorizer, Tfidf Vectorizer, each with uni, bi, and trigrams.

In the above image, ‘Ac’ represents the accuracy score and ‘cv_1’ means CountVectorizer with unigrams. Both PassiveAggressiveClassifier (Tfidf-bigrams) and LinearSVC(Tfidf-trigrams) gives the highest accuracy of 83.5714%. But the former takes only 4.5 seconds to be trained while the latter takes 18 seconds. In terms of cross-validation mean score and standard deviation, LinearSVC seems better. It is worth noting that our dataset is imbalanced- with more positive and negative tweets than neutral tweets. So, in this case, the accuracy scores may lead us to wrong conclusions. To avoid that, we also used Random Forest Classifier with balanced class weights. And this classifier also performs reasonably well with an average accuracy score of 74.94% and highest accuracy of 78.21%.

Implementing Gensim models

We have used bag-of-words models(CV and Tfidf) for numerical representation of text documents. This is a quite simple method, mathematically. In this method, the semantic relation between words is not preserved. Say, there are four words in a sentence: king, queen, woman, man. In BoW models, each of these words is assigned ids uniquely. So the word ‘king’ may be nearer to (in terms of the vector values )’woman’ than ‘queen’. So we need a sophisticated model which preserves the relationships between words, thus giving a better mathematical representation of the text. Doc2Vec (an extension of Word2Vec) models do this for us.

It would be better if the words are semantically related as in above figure rather than being assigned some random ids

Doc2Vec algorithm is used from Gensim library in python. Gensim is a free Python library designed to automatically extract semantic topics from documents, as efficiently as possible. We implement three algorithms of doc2vec- DBOW(Distributrb Bag of Words), DMC(Distributed Memory Concatenated), DMM(Distributed Memory Mean) and a few combinations of these three. Later, we use all of these models to train the same classifiers as we have done above using sklearn. The steps to be done for each of these algorithms are as follows- concatenate all the datasets(train, validation, and test), give indices(labels) to all the tweets(can be random labels), train the model(DBOW,DMC or DMM) , get the vectors from each model.

x_all = pd.concat([x_train,x_val,x_test])

This first step of concatenation is optional(i.e., we can use only training dataset) but it is suggested to concatenate to attain better results. Then we label the text with ‘d2v_index’ where ‘index’ is the tweet index.

def label_tweets(tweets,label):

result = []

prefix = label

i=0

for i, t in zip(tweets.index, tweets):

t=str(t)

result.append(LabeledSentence(t.split(), [prefix + '_%s' % i]))

return result

x_all_w2v = label_tweets(x_all,'d2v')#x_all is obtained above by conatenating.

Then, we train the three models.

DBOW

cpu_count = multiprocessing.cpu_count()

model_dbow = Doc2Vec(dm=0,size=100,negative=5,min_count=2,alpha=0.065,workers=cpu_count,min_alpha=0.065)

model_dbow.build_vocab([each for each in tqdm(x_all_w2v)]) for epoch in range(30):

model_dbow.train(utils.shuffle([each for each in tqdm(x_all_w2v)]),total_examples=len(x_all_w2v),epochs=1)

model_dbow.alpha-=0.002

model_dbow.min_alpha = model_dbow.alpha train_vecs_dbow = get_d2v_vectors(model_dbow,x_train,100)

val_vecs_dbow = get_d2v_vectors(model_dbow,x_val,100)

In the first line, we obtain the number of CPUs for parallel processing. After defining the model(in the second line), we build our vocabulary using thee labeled tweets (x_all_w2v). Later, we train the model, adjusting our learning rate in each iteration. Please read this documentation to know more about the parameters of Doc2Vec function.

The other two models are trained in the same manner, after defining the parameters specific to each of them.

DMC

cpu_count = multiprocessing.cpu_count()

model_dmc = Doc2Vec(dm=1,dm_concat=1,size=100,negative=5,min_count=2,alpha=0.065,workers=cpu_count,min_alpha=0.065)

model_dmc.build_vocab([each for each in tqdm(x_all_w2v)]) for epoch in range(30):

model_dmc.train(utils.shuffle([each for each in tqdm(x_all_w2v)]),total_examples=len(x_all_w2v),epochs=1)

model_dmc.alpha-=0.002

model_dmc.min_alpha = model_dmc.alpha train_vecs_dmc = get_d2v_vectors(model_dmc,x_train,100)

val_vecs_dmc = get_d2v_vectors(model_dmc,x_val,100)

DMM

cpu_count = multiprocessing.cpu_count()

model_dmm = Doc2Vec(dm=1,dm_mean=1,size=100,negative=5,min_count=2,alpha=0.065,workers=cpu_count,min_alpha=0.065)

model_dmm.build_vocab([each for each in tqdm(x_all_w2v)]) for epoch in range(30):

model_dmm.train(utils.shuffle([each for each in tqdm(x_all_w2v)]),total_examples=len(x_all_w2v),epochs=1)

model_dmm.alpha-=0.002

model_dmm.min_alpha = model_dmm.alpha train_vecs_dmm = get_d2v_vectors(model_dmm,x_train,100)

val_vecs_dmm = get_d2v_vectors(model_dmm,x_val,100)

We save all the three models to eliminate re-training for any future use.

model_dbow.save('d2v_model_dbow.doc2vec')

model_dmm.save('d2v_model_dmm.doc2vec')

model_dmc.save('d2v_model_dmc.doc2vec')

#AFTER THE FIRST TIME DON'T RETRAIN ALL THE MODELS,JUST LOAD THESE #like mod = model_dmm.load()

After training, each of these models we also extract vectors using get_d2v_vectors() function as below(Remember? This is our main objective to use Gensim models- to get a vectorized representation of the text data). The below function is called after training the models(in the above snippets).

def get_d2v_vectors(model,corpus,size):

vecs = np.zeros((len(corpus),size))

n=0

for i in corpus.index:

prefix='d2v_'+str(i)

vecs[n] = model.docvecs[prefix]

n+=1

return vecs

We are ready with the three vector representations using doc2vec models. (train_vectors_dbow, train_vectors_dmc, train_vectors_dmm). Before using them with classifiers, there’s another useful step to perform- concatenate these vectors(for instance, DBOW+DMM). This step is advised by the authors of Doc2vec for better results. We use numpy.append() function to concatenate these vectors.

#FUNCTION TO CONCAT 2-MODELS VECTORS..NO SPECIAL TRAINING IS REQUIRED

def get_concat_vectors(model1,model2,corpus,size):

vecs = np.zeros((len(corpus),size))

n=0

for i in corpus.index:

prefix='d2v_'+str(i)

vecs[n] = np.append(model1.docvecs[prefix],model2.docvecs[prefix])

n+=1

return vecs

We concatenate DBOW with DMM and DBOW with DMC.

train_vecs_dbow_dmm = get_concat_vectors(model_dbow,model_dmm,x_train,200)

val_vecs_dbow_dmm = get_concat_vectors(model_dbow,model_dmm,x_val,200) train_vecs_dbow_dmc = get_concat_vectors(model_dbow,model_dmc,x_train,200)

val_vecs_dbow_dmc = get_concat_vectors(model_dbow,model_dmc,x_val,200)

Now we use the same sklearn algorithms for classification purpose using these five vectors in a similar way(as we have done with CV and Tfidf).

vecs = [(train_vecs_dbow,val_vecs_dbow),(train_vecs_dmc,val_vecs_dmc),(train_vecs_dmm,val_vecs_dmm),(train_vecs_dbow_dmm,val_vecs_dbow_dmm),(train_vecs_dbow_dmc,val_vecs_dbow_dmc)]

classifiers = [BernoulliNB(),LogisticRegression(),LinearSVC(),AdaBoostClassifier(),RidgeClassifier(),PassiveAggressiveClassifier(),Perceptron()]

gensim_names = ['DBOW','DMC','DMM','DBOW+DMM','DBOW+DMC']

data=[]

for train_vecs,val_vecs in vecs:

for clf in classifiers:

clf.fit(train_vecs,y_train)

ac = clf.score(val_vecs,y_val)

data.append([gensim_names[j],clf_names[i],ac,end-start])

d = pd.DataFrame(data,columns=['Model','Classifier','Ac','Time'])

d.to_csv('gensim_all_clfs.csv')

Only a few important lines of code are given above. See this.

Let’s have a look at the results.

Results using Gensim models

Logistic Regression(DBOW), LinearSVC(DBOW), Logistic Regression (DBOW +DMC ) and Linear SVC(DBOW+DMC) gives the highest accuracy of 65.35%. Taking training time into consideration, Logistic Regression(DBOW) is the best one among these(with 0.52 seconds). RidgeClassifier(DBOW) also shows good performance with 64.2% accuracy and just 0.007 seconds for training. DBOW model outperforms the other two models with a mean accuracy of (across all the classifiers) 60.4%. The Gensim models show slightly poorer performance than that of CV and Tfidf models.

All in all, Passive Aggressive Classifier(Tfidf-bigrams) performs the best with an accuracy of 83.57%. So let’s evaluate it using test data.

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.linear_model import PassiveAggressiveClassifier

from sklearn.pipeline import make_pipeline

from sklearn.metrics import confusion_matrix

from sklearn.metrics import accuracy_score,precision_score,recall_score

import time

b=2

st = time.time()

tfidf = TfidfVectorizer(ngram_range=(1,b))

model = make_pipeline(tfidf,PassiveAggressiveClassifier())

model.fit(x_train.values.astype('U'),y_train.values.astype('U'))##

labels = model.predict(x_test.values.astype('U'))### cm = confusion_matrix(y_test.values.astype('U'),labels)

ac = accuracy_score(y_test.values.astype('U'),labels)

pr = precision_score(y_test.values.astype('U'),labels,average=None)

rc = recall_score(y_test.values.astype('U'),labels,average=None)

en = time.time()

print(b,"-gram",ac,pr.mean(),rc.mean(),en-st)

When this model is evaluated on test data it gives an accuracy of 80.42%.

Summary

Tweets are collected using Tweepy library followed by the selection of useful features to my task and conversion to CSV. Then, data cleaning is performed like removing URLs, retweet symbols, username-tags, and hashtags. SentiWordNet lexicon is used to label the sentiment of the tweets. Steps like Stop words removal, Lemmatizing, Stemming are performed on the text data. Later, WordCloud is used for Data Visualization. After splitting the data, Count Vectorizer and Tfidf Vectorizer are used for mathematical representation of the text. Then, nine classification algorithms are implemented on the vectors obtained previously. Later, doc2vec models (DBOW, DMC, DMM) are trained and used vectors (obtained through these models)for classification purpose as these models preserve semantic relationships between words. Finally, the best model is evaluated using the test data.

Any suggestions for the improvement of this project are welcome. Thank you for reading!

References

1 https://towardsdatascience.com/another-twitter-sentiment-analysis-with-python-part-6-doc2vec-603f11832504

2 https://bhaskarvk.github.io/2015/01/how-to-use-twitters-search-rest-api-most-effectively./

3 https://medium.com/analytics-vidhya/twitter-sentiment-analysis-for-the-2019-election-8f7d52af1887

4https://chrisalbon.com/machine_learning/trees_and_forests/handle_imbalanced_classes_in_random_forests/