A Guide to Text Classification and Sentiment Analysis
By: Abhijit Roy
Tags: Sentiment Analysis, Lstm, Text Classification

A Guide to Text Classification and Sentiment Analysis

Motivation: Text Classification and sentiment analysis is a very common machine learning problem and is used in a lot of activities like product predictions, movie recommendations, and several others. Currently, for every machine learner new to this field, like myself, exploring this domain has become very important. After exploring the topic, I felt, if I share my experience through an article, it may help some people trying to explore this field. So, I will try to build the whole thing from a basic level, so this article may look a little long, but it has some parts you can skip if you want.

Text classification problems like sentimental analysis can be achieved in a number of ways using a number of algorithms. These are majorly divided into two main categories:

A bag of Word model: In this case, all the sentences in our dataset are tokenized to form a bag of words that denotes our vocabulary. Now each individual sentence or sample in our dataset is represented by that bag of words vector. This vector is called the feature vector. For example, ‘It is a sunny day’, and ‘The Sun rises in east’ are two sentences. The bag of words would be all the words in both the sentences uniquely. The second method is based on a time series approach: Here each word is represented by an Individual vector. So, a sentence is represented as a vector of vectors.

This is the very basic difference between the two approaches. Now, let’s see in detail.

For the purpose, we will use, the IMDB review dataset. It has 50,000 reviews and their corresponding sentiments marked as “Positive” and “Negative”. So, first, we need to check and clean our dataset.

Initial Dataset

from bs4 import BeautifulSoup def strip_html(text):

soup = BeautifulSoup(text, "html.parser")

return soup.get_text() import re def remove_between_square_brackets(text):

return re.sub('\[[^]]*\]', '', text) def denoise_text(text):

text = strip_html(text)

text = remove_between_square_brackets(text)

return text def remove_special_characters(text, remove_digits=True):

pattern=r'[^a-zA-z0-9\s]'

text=re.sub(pattern,'',text)

return text

def Convert_to_bin(text, remove_digits=True):

if text=='positive':

text= 1

else:

text=0

return text

These are the preprocessing required to clean the reviews part and the Convert_to_binary is used to convert “Positive” sentiment to 1 and “Negative” to 0. So, after preprocessing we obtain:

After PreProcessing

If we count plot our target class division.

We have 25k positive sentiments and 25k negative sentiments. So, let’s split up the data.

X=df['review'].values

Y=df['sentiment'].values

X_train, X_test, Y_train, Y_test= train_test_split(X,Y, test_size=0.3)

The First Approach

Vectorizers: Bag of words

In this method, we create a single feature vector using all the words in the vocabulary, that we obtain from tokenizing the sentences in the train sets. This feature vector is used to represent all the reviews in our set. Each word is basically regarded as a feature. So, the number of features is equal to the number of unique words in the vocabulary. Now, each sentence or review is regarded as a sample or record. If the word is present in that sample it has some values for the word which is regarded as the feature, and if the word is not present it is zero. So, if ‘It is a sunny day’, and ‘The Sun rises in east’ are two sentences. and their bag will have 10 words which will be the feature vector size. Say vector is: [‘It’, ‘is’, ‘a’, ‘sunny’, ‘day’, ‘The’, ‘Sun’, ‘rises’, ‘in’, ‘east’]. So, the first sentence is represented by [a,b,c,d,e,0,0,0,0,0], a,b,c,d,e are values depending on the scheme we use

So, each sample has the same feature set size which is equal to the size of the vocabulary. Now, the vocabulary is basically made of the words in the train set. All the samples of the train and test set are transformed using this vocabulary only. So, there may be some words in the test samples which are not present in the vocabulary, they are ignored.

They form very sparse matrices or feature sets. Similar to a normal classification problem, the words become features of the record and the corresponding tag becomes the target value. If we consider as a dataset, the samples or reviews will be the rows or records, the feature set of each record, or the feature columns corresponding to each record will be equal to the size of the vocabulary, where each word will be a feature. So, it is actually like a common classification problem with the number of features being equal to the distinct tokens in the training set.

This can be done in two ways:

Count Vectorizer: Here the count of a word in a particular sample or review. The count of that word becomes the value of the corresponding word feature. If a word in the vocab does not appear in the sample its value is 0. So basically if there are 10 sentences in the train set, Those 10 sentences are tokenized to create the bag. Say, the created bag length is 100 words. So, we will form a vector of length 100. For a particular sample, if there are 15 words, where there is one word appearing three times in that sample, the value corresponding to that word becomes 3 for that sample and so on. TF-IDF Vectorizer: It is a better approach. It calculates two things term frequency and inverse document frequency. Term frequency= No. of times the word appears in the sample. and IDF = log ( number of times the word appears in the sample/number of times the word appears in the whole document). This helps to note some differences like the word “The” appears with the same freq in almost all sentences while special words carrying significance like “good” don’t. So, these TF and IDF values of each word for a specific sample are multiplied to obtain the feature vectors for that sample.

We will be using scikit-learn’s feature extraction libraries here. But one thing to notice here is, this can also be done using TensorFlow's Tokenizer function. It can be done in mainly three ways using tokenizer:

Binary: X = tokenizer.sequences_to_matrix(x, mode=’binary’): In this case, the value of a word feature is 1 if the word is present in the sample else zero. Count: X = tokenizer.sequences_to_matrix(x, mode=’count’) : In this case, the value is the number of times a word appear in the sentence. TF-IDF: X = tokenizer.sequences_to_matrix(x, mode=’tfidf’): In this, we consider the TF of the word in the sample and IDF of the word in the sample with respect to the occurrence of the word in the whole document.

Count Vectorizer: Code Implementation

from sklearn.feature_extraction.text import CountVectorizer

vec = CountVectorizer()

vec.fit(X_train)

x_train=vec.transform(X_train)

x_test=vec.transform(X_test)

Now, if we notice, the vector is fit only to X_train. Here is where the vocabulary is formed. So vocabulary contains only the words in the train set. Then we transform on both train and test set.

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Dense

from tensorflow.keras.optimizers import Adam

model = Sequential()

model.add(Dense(16, input_dim=x_train.shape[1], activation='relu'))

model.add(Dense(16, activation='relu'))

model.add(Dense(1, activation='sigmoid'))

model.compile(loss='binary_crossentropy',optimizer='Adam',metrics=['accuracy'])

This is our model for the count vectorization approach. We can see that the input dimension is of size equal to the number of columns for each sample which is equal to the number of words in our vocabulary. The output is 1 i.e, sentiment Positive or Negative.

The model gives an accuracy of 87.3%

TF-IDF Vectorizer: Code Implementation

from sklearn.feature_extraction.text import TfidfVectorizer

vec = TfidfVectorizer()

vec.fit(X_train)

x_train=vec.transform(X_train)

x_test=vec.transform(X_test)

Now, here we call the TF-IDF vectorizer and the data is transformed. Here we use a logistic regression model.

from sklearn.linear_model import LogisticRegression

lr = LogisticRegression()

lr.fit(x_train, Y_train)

This model gives an accuracy of 89.4%

The Second Approach

Word Embedding: Time series approach

In this method, the words are individually represented as a vector. In the case of the bag of words, all of the words in the vocabulary made up a vector. Say, there are 100 words in a vocabulary, so, a specific word will be represented by a vector of size 100 where the index corresponding to that word will be equal to 1, and others will be 0.

So, Each sample having a different number of words will basically have a different number of vectors, as each word is equal to a vector. Now, to feed a model we will need to have the same dimension for each sample, and as a result, padding is needed to make the number of words in each sample equal to each other.

Basically in the bag of words or vectorizer approach, if we have 100 words in our total vocabulary, and a sample with 10 words and a sample with 15 words, after vectorization both the sample sizes would be an array of 100 words, but here for the 10 words it will be a (10 x 100) i.e, 100 length vector for each of the 10 words and similarly for 15th one size will be (15 x 100). So, we need to find the longest sample and pad all others up to match the size.

We can do this in some ways:

One-Hot encoding: It is just taking the size of the vocabulary and making an array of that size with 0’s at all indices and 1 at only the index of the word, as we have discussed above. But these things provide us with very little information and create a very sparse matrix.

The Second Choice is the word embeddings.

The one-hot encoder is a pretty hard-coded approach. It is of a very high dimension and sparse with a very low amount of data. Embedding is a way to create a dense vector representation. It is of a lower dimension and helps to capture much more information. It more like captures the relationships and similarities between words using how they appear close to each other. For example, the king, queen, men, and women will have some relations.

Say, we are having 10k words are being embedded in a 300-dimensional embedding space. To do this, we declare the number of nodes in the embedding layer=300. Now, each word of the 10k words enters the embedding layer. Each of the words will be placed in a 300-dimensional plane based on their similarities with one another which is decided by several factors, like the order in which the words occur. Now, being placed in 300 Dimensional planes the words will have a 300 length tuple to represent it which are actually the coordinates of the point on the 300-dimensional plane. So, this 300-dimensional tuple becomes the new feature set or representing a vector for the word.

So, the vector for the word decreased from 10k to 300. The tuples serve as feature vectors between two words and the cosine angle between the vectors represents the similarity between the two words.

We can do this in two ways:

Using our own embeddings Using pre-trained embeddings

Making our own embedding

Now, for the embedding, we need to send each sample through an embedding layer first then move to make them dense using embedding. These embedding layers see how the words are used, i.e, it tries to see if two words always occur together or are used in contrast. After judging all these factors the layer places the word in a position one the n-dimensional embedding space.

Using pre-trained embedded matrices

We can use pre-trained word embeddings like word2vec by google and GloveText by Standford. They are trained on huge corpora with billions of examples and words. Now, they have billions of words we have only say, a 10k so, training our model with a billion words will be very inefficient. We need to just select out our required word’s embeddings from their pre-trained embeddings.

Now, How are these embeddings found?

For google’s word2vec implementations, there are two ways:

Continous bag of words Skip-Gram.

Both of these algorithms actually use a Neural Network with a single hidden layer to generate the embedding.

For CBOW, the context of the words, i.e, the words before, and after the required words are fed to the neural network, and the model is needed to predict the word.

For the Skip-Gram, the words are given and the model has to predict the context words.

In both cases, the feature vectors or encoded vectors of the words are fed to the input. The output has a softmax layer with a number of nodes equal to the vocabulary size, which gives the percentage of prediction for each word, i.e, it tells what is the probability that the required word is the word, the node in the softmax layer is representing. Though we don’t use the output layer actually.

We go for the weight matrix produced in the hidden layer. The number of nodes in the hidden layer is equal to the embedding dimension. So, say if there are 10k words in vocabulary and 300 nodes in the hidden layer, each node in the hidden layer will have an array of weights of the dimension of 10k for each word after training.

Because the neural network units work on

y=f(w1x1+w2x2+……….wnxn)

Here x1, x2…… xn are the words and so n= number of words in vocabulary=10k for our case.

So for 10k x’s, there will be 10k w’s. Now for 1 node, there is a 10k length-weight matrix. For 300 combined we have a matrix of 300 x 10k weights. Now, if we concatenate, we will have 300 rows and 10k columns. Let’s transpose the matrix. We will get 300 columns and 10k rows. Each row represents a word, and the 300 column values represent a 300 length-weight vector for that word.

This weight vector is the obtained embedding of length 300.

Let’s look at implementations:

Training Own Embedding

from keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer(num_words=10000)

tokenizer.fit_on_texts(X_train)

Num_words indicates the size of the vocabulary

x_train = tokenizer.texts_to_sequences(X_train)

x_test = tokenizer.texts_to_sequences(X_test)

1) tokenize.fit_on_text() →> Creates the vocabulary index based on word frequency. For example, if you had the phrase “My dog is different from your dog, my dog is prettier”, word_index[“dog”] = 0, word_index[“is”] = 1 (‘dog’ appears 3 times, ‘is’ appears 2 times)

2) tokenize.text_to_sequence() →> Transforms each text into a sequence of integers. Basically, if you had a sentence, it would assign an integer to each word from your sentence. You can access tokenizer.word_index() (returns a dictionary) to verify the assigned integer to your word.

vocab = len(tokenizer.word_index) + 1

from keras.preprocessing.sequence import pad_sequences

maxlen = 100

x_train = pad_sequences(x_train, padding='post', maxlen=maxlen)

x_test = pad_sequences(x_test, padding='post', maxlen=maxlen)

We are padding all sentences to a length of max length 100.

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1D

from tensorflow.keras.optimizers import Adam

emb_dim=100

model= Sequential()

model.add(Embedding(input_dim=vocab, output_dim=emb_dim, input_length=maxlen))

model.add(MaxPool1D())

model.add(Dense(16,activation="relu"))

model.add(Dense(16,activation="relu"))

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])

Max pool layer is used to pick out the best-represented features to decrease sparsity.

Model summary

This model gives an accuracy of 77.4%

Using Pre-trained Embeddings

We will be using Standford's Glove embedding which is trained over 6Billion words.

It has four files each with a different embedding space, we will be using the 50d one, which is a 50-Dimensional Embedding space.

['glove.6B.100d.txt', 'glove.6B.200d.txt', 'glove.6B.50d.txt', 'glove.6B.300d.txt']

Implementations:

import numpy as np

from tensorflow.keras.preprocessing.text import Tokenizer

tokenizer = Tokenizer(num_words=10000)

tokenizer.fit_on_texts(X_train)

x_train = tokenizer.texts_to_sequences(X_train)

x_test = tokenizer.texts_to_sequences(X_test)

from keras.preprocessing.sequence import pad_sequences

maxlen=100

x_train = pad_sequences(x_train, padding='post', maxlen=maxlen)

x_test = pad_sequences(x_test, padding='post', maxlen=maxlen)

So, let’s see how to extract the embedding we require from the given embedding file.

emb_dim=50

vocab=len(tokenizer.word_index)+1

emb_mat= np.zeros((vocab,emb_dim))

#Initializing a zero matrix for each word, they will be compared to have their final embedding with open(file_path+'glove.6B.50d.txt') as f:

for line in f:

word, *emb = line.split()

if word in tokenizer.word_index:

ind=tokenizer.word_index[word]

emb_mat[ind]=np.array(emb,dtype="float32")[:emb_dim]

This is an extractor for the task, so we have the embeddings and the words in a line. So, we just compare the words to pick out the indices in our dataset. Take the vectors and place them in the embedding matrix at an index corresponding to the index of the word in our dataset.

We have used an *emb because the embedding matrix is variant in size.

So, we here have a feature set with a vocabulary of 10k words and each word represented by a 50 length tuple embedding which we obtained from the Glove embedding.

Next, we build our model.

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1D

from tensorflow.keras.optimizers import Adam

emb_dim=50

maxlen=100

model= Sequential()

model.add(Embedding(input_dim=vocab, output_dim=emb_dim,weights=[emb_mat], input_length=maxlen,trainable=False))

model.add(MaxPool1D())

model.add(Dense(16,activation="relu"))

model.add(Dense(16,activation="relu"))

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])

Model Summary

This model gives an accuracy of 67% probably due to the decreased embedding size. The larger the embedding size more the information contained.

Now, how these approaches are beneficial over to the bag of words model? As we can see these Bag of Words models just saw how a word behaves in the document, i.e, what can we tell about the frequency of occurrence of the word or any pattern in which the word occurs? While these approaches also take into consideration the relationship between two words using the embeddings.

Use of Convolutional Neural Networks for classification

In mathematics (in particular, functional analysis) convolution is a mathematical operation on two functions (f and g) that produces a third function expressing how the shape of one is modified by the other. The term convolution refers to both the result function and to the process of computing it.

So, Convolutional is best for extracting special features and behavior of feature values from the 2D pixels of images. Convolutional layers have a set of kernels which helps to extract several important features from the data samples. Now here, in case of text classifications, our feature matrices are 1Dimensional. So, here Conv1D is used. Basically it moves as a sliding window of size decided by the user. We have chosen 5.

Now, initially after embedding, we get 100 Dimensional embeddings. Next using 1D convolutions we try to make our feature set smaller and let the feature set discover the best features relations for the classification. The max-pooling layer also helps to pick the features or words which have the best performance.

The convolutional layer is always used after an embedding layer after it provides its embedded feature vectors.

Implementations:

from tensorflow.keras.models import Sequential

from tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1D,Conv1D

from tensorflow.keras.optimizers import Adam

emb_dim=100

model= Sequential()

model.add(Embedding(input_dim=vocab, output_dim=emb_dim,

input_length=maxlen))

model.add(Conv1D(64, 5, activation='relu'))

model.add(MaxPool1D(5))

model.add(Conv1D(128, 5, activation='relu'))

model.add(MaxPool1D(5))

model.add(Dense(16,activation="relu"))

model.add(Dense(1, activation='sigmoid'))

model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])

Model Summary

Here we have trained our own embedding matrix of dimension 100. We have used two convolutional layers with 64 and 128 kernel filters respectively. We have kept our sliding window =5.

This model has given a test accuracy of 77%

Use of Recurrent Neural Networks for classification

Why Recurrent Neural Networks?

Until now we have tried to extract some features from all the words in a sample at a time. So, all of them are non-temporal approaches. Now, let’s see how a person will judge a sentiment. He/she will not only consider what were the words used, but humans will also consider how they are used, that is, in what context, and what are the preceding and succeeding words? So, until now we have focused on what were the words used only, so, now let’s look at the other part of the story.

So, for this part, we need a Recurrent neural network to give a memory to our models. If we think about telling something about someone's statements, we will generally listen to the whole statement word by word and then make a comment. This is what the Recurrent Neural networks will accomplish. It will look at each word in a temporal manner one by one and try to correlate to the context using the embedded feature vector of the word.

We know RNN suffers from the vanishing and exploding gradient problem we will be using LSTM.

LSTM operates on two things a hidden state that is sent from a previous timestamp and a cell state that actually maintains the weight neutralizing the vanishing gradient effect.

The LSTM layer basically has 4 components: A Forget gate, An input gate, a cell state, and an output gate.

Now, let’s talk a bit about the working and dataflow in an LSTM, as I think this will help to show how the feature vectors are actually formed and what it looks like.

LSTM Equations

These are the set of equations LSTM operates upon.

LSTM provides a feature set on the last timestamp for the dense layer, to use the feature set to produce results. We can see the above equations are the equations for the Gates of LSTM. Here, each gate acts as a neural network individually. So, they have their individual weight matrices that are optimized when the recurrent network model is trained. Using these weight matrices only the gates learn their tasks, like which data to forget and what part of the data is needed to be updated to the cell state. So, the gates optimize their weight matrices and decide the operations according to it.

Now, let's see the use.

Say we have a 100-dimensional vector space. a batch size of 16, each sample length = 10. and the number of nodes in each layer= 64.

INPUT SIZE = batch_size * Embedding so, here it is 16 x 100 matrix = x(t)

The timestamp 0 that is the first word of every sample or record enters.

The x0 represents the first word of the samples, x1 represents second, and so on. So, each time 1 word from 16 samples and each word is represented by a 100 length vector. So, the size of each input is (16 x 100).

PREVIOUS HIDDEN STATE (0 vector for tiemstamp 0) = Batch size x Hidden Units So, Here it is 16 x 64 matrix.= h(t-1)

After concatenation, the matrix formed h(t-1)= 16 x 64 and x(t)= 16 x 100

So the h(t-1) + x(t) matrix is sent to all gates f(t) , u(t) and o(t) .