Twitter Hate Speech Sentiment Analysis
By: Mustafa Abusalah
Tags: Machine Learning, Ensemble, Sentiment Analysis, Twitter, Corpus Linguistics

Training and Testing Data

Training Data provided on the challenge page is about 32k records and decomposed into tweets and labels, that indicates in 1 or 0 whether the tweet contains abuse or not. The actual testing or judging data consists of about 17k records.

Training and Testing data required some cleaning as they have special characters, urls or html tags, hash tags mentions and etc. To start let us import the necessary libraries first

from sklearn import model_selection, preprocessing, metrics, svm, ensemble

from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer

import pandas as pd, numpy, string

from nltk.tokenize import WordPunctTokenizer

from nltk.stem import PorterStemmer

from bs4 import BeautifulSoup

#Remove Special Charactors

import re

Data-sets preparation and Cleaning

For the purpose of this exercise, I have removed the special characters, converted words to lower case and made sure there are no null rows. Also I have created a data frame with two columns, tweets and labels to represent the csv files as you will see later. I have also applied Porter Stemmer in order to reduce words inflection. And I removed the numbers and extra white space. Part of the code in the below block was reused from this article.

#Import Training and Testing Data

train = pd.read_csv('train.csv')

print("Training Set:"% train.columns, train.shape, len(train))

test = pd.read_csv('test_tweets.csv')

print("Test Set:"% test.columns, test.shape, len(test)) #Tokenize words in order to clean and stem

tok = WordPunctTokenizer()

# patterns to remove html tags numbers and special Characters

pat1 = r'@[A-Za-z0-9]+'

pat2 = r'https?://[A-Za-z0-9./]+'

combined_pat = r'|'.join((pat1, pat2))

porter=PorterStemmer() def tweet_cleaner(text):

soup = BeautifulSoup(text, 'lxml')

souped = soup.get_text()

stripped = re.sub(combined_pat, '', souped)

try:

clean = stripped.decode("utf-8-sig").replace(u"\ufffd", "?")

except:

clean = stripped

letters_only = re.sub("[^a-zA-Z]", " ", clean)

lower_case = letters_only.lower()

# During the letters_only process two lines above, it has created unnecessay white spaces,

# Tokenize and join together to remove unnecessary white spaces

words = tok.tokenize(lower_case)

#Stemming

stem_sentence=[]

for word in words:

stem_sentence.append(porter.stem(word))

stem_sentence.append(" ")

#Rejoin the words back to create the cleaned tweet

words="".join(stem_sentence).strip()

return words

Data shape:

Training Set: (31962, 3) 31962

Test Set: (17197, 2) 17197

Now create an array of cleaned tweets for both training and testing data, then create a dataframe for both:

nums = [0,len(train)]

clean_tweet_texts = []

for i in range(nums[0],nums[1]):

clean_tweet_texts.append(tweet_cleaner(train['tweet'][i]))

nums = [0,len(test)]

test_tweet_texts = []

for i in range(nums[0],nums[1]):

test_tweet_texts.append(tweet_cleaner(test['tweet'][i])) #Dataframes to represent the testing and training samples

train_clean = pd.DataFrame(clean_tweet_texts,columns=['tweet'])

train_clean['label'] = train.label

train_clean['id'] = train.id

test_clean = pd.DataFrame(test_tweet_texts,columns=['tweet'])

test_clean['id'] = test.id

For the sake of checking the performance or our classifier we need to split the training data into testing and validation sets, I used model selection library for this purpose. Then I have encoded the labels to convert them to numpy arrays

# split the dataset into training and validation datasets

train_x, valid_x, train_y, valid_y = model_selection.train_test_split(train_clean['tweet'],train_clean['label'])

# label encode the target variable

encoder = preprocessing.LabelEncoder()

train_y = encoder.fit_transform(train_y)

valid_y = encoder.fit_transform(valid_y)

TFIDF: Term Frequency Inverse Document Frequency

Now we need to identify the importance of words in the corpus (Data-set) to feed the classifier. The best method to use is TFIDF.

TFIDF score represents the relative importance of a term (word) in the document (tweet) and the entire corpus (data-set). TFIDF score is composed by two terms: the first computes the normalized Term Frequency (TF), the second term is the Inverse Document Frequency (IDF), computed as the logarithm of the number of the documents in the corpus divided by the number of documents where the specific term appears.

I have applied TFIDF on a word level, that creates a matrix representing tfidf scores of every term in different tweets. You may also apply tfidf on ngram base or character base.

# word level tf-idf

tfidf_vect = TfidfVectorizer(analyzer='word', token_pattern=r'\w{1,}', stop_words='english', max_features=100000)

tfidf_vect.fit(train_clean['tweet'])

xtrain_tfidf = tfidf_vect.transform(train_x)

xvalid_tfidf = tfidf_vect.transform(valid_x)

You may use Bag or Words to measure the words importance, or any other tool you believe it would be better, for bag of words check the below:

#Bag of Words

from sklearn.feature_extraction.text import CountVectorizer

bow_vectorizer = CountVectorizer(max_df=0.90, min_df=2, max_features=1000, stop_words='english')

# bag-of-words feature matrix

bow = bow_vectorizer.fit_transform(train_clean['tweet'])

xtrain_bow = tfidf_vect.transform(train_x)

xvalid_bow = tfidf_vect.transform(valid_x)

F1 Score

Now to test our model we need to check the f1 score to measure the accuracy, which is required in the challenge and your results will be assessed based on this method. f1 score considers both the precision p and the recall r of the test to compute the score: p is the number of correct positive results divided by the number of all positive results returned by the classifier, and r is the number of correct positive results divided by the number of all relevant samples (all samples that should have been identified as positive).

F1 Score

The f1 score is the harmonic average of the precision and recall, where an f1 score reaches its best value at 1 (perfect precision and recall) and worst at 0.

#Return the f1 Score

def train_model(classifier, feature_vector_train, label, feature_vector_valid):

# fit the training dataset on the classifier

classifier.fit(feature_vector_train, label)



# predict the labels on validation dataset

predictions = classifier.predict(feature_vector_valid) return metrics.f1_score(valid_y,predictions)

Actual Data Predictions

In the below method we do the assessment for the real testing data:

#Return Prediction Labels

def train_model_Pred(classifier, feature_vector_train, label, feature_vector_valid):

# fit the training dataset on the classifier

classifier.fit(feature_vector_train, label)



# predict the labels on validation dataset

predictions = classifier.predict(feature_vector_valid)



return predictions

Ensemble Model

Scikit-Learn provides a very powerful prediction library called Ensemble. ensembling increase accuracy on a variety of Machine Learning tasks. As the training corpus provided for this challenge is biased, only 7% of the corpus contains hate speech, we need to use a method that could reduce the bias in the training data.

To overcome the bias in the corpus, I used Extremely Randomized Tree to create a diverse set of classifiers by introducing randomness in the classifier construction. The prediction of the ensemble is given as the averaged prediction of the individual classifiers.

As other classifiers, forest classifiers have to be fitted with two arrays: a sparse or dense array X of size [xtrain_tfidf, train_y] holding the training samples, and an array Y of size [xvalid_tfidf] holding the target values (class labels) for the training samples. Note that I have set the n_estimators to 300 to get higher accuracy results, the default is 10. Saying so the processing time will higher. But keep in mind the Random Forests are designed for large set of data.

# Extremely Randomized Trees Classifier on Word Level TF IDF Vectors

accuracy = train_model(ensemble.ExtraTreesClassifier(n_estimators=300), xtrain_tfidf, train_y, xvalid_tfidf)

print("Extremely Randomized Trees, WordLevel TF-IDF: ", accuracy)

The accuracy based on f1 score for the training and validation data was:

Extremely Randomized Trees, WordLevel TF-IDF: 0.6965517241379311

Now let us make the real work and work on the real test corpus, first create TFIDF vectors:

#Now working with Real challenge Data

train_x=train_clean['tweet']

valid_x=test_clean['tweet']

train_y=train_clean['label']

# label encode the target variable

encoder = preprocessing.LabelEncoder()

train_y = encoder.fit_transform(train_y)

tfidf_vect = TfidfVectorizer(analyzer='word', token_pattern=r'\w{1,}', stop_words='english', max_features=100000)

tfidf_vect.fit(train_clean['tweet'])

tfidf_vect.fit(test_clean['tweet'])

xtrain_tfidf = tfidf_vect.transform(train_x)

xvalid_tfidf = tfidf_vect.transform(valid_x)

Send the Training and Testing Data to the Classifier:

# Extremely Randomized Trees on Word Level TF IDF Vectors

accuracy = train_model_Pred(ensemble.ExtraTreesClassifier(n_estimators=300), xtrain_tfidf, train_y, xvalid_tfidf)

print("Extremely Randomized Trees, WordLevel TF-IDF: ", accuracy)

Now the accuracy array is set:

Extremely Randomized Trees, WordLevel TF-IDF: [0 1 0 ... 0 0 0]

Let us convert this to Dataframe and export to csv file for submission:

#If you want to check your results manually you may include the tweet:d={'id':test['id'],'Tweet':valid_x,'label':accuracy}

d={'id':test['id'],'label':accuracy}

df=pd.DataFrame(data=d)

df.to_csv("test_predictions.csv", index=False)

Now we are done and once the code and result was submitted to the competition, I was able to score: 0.7346278317

Saying so, I have done some previous attempts using oversampling methods Random Over Sampling ROS and Synthetic Minority Over-sampling Technique SMOTE with Linear Regression classifier, I will discuss over sampling in a different article. In general Ensemble Extremely Randomized Trees was the best method used on the original cleaned data and over came Linear Regression even after Over Sampling.

Full working code is available on github here.