A Guide to Text Classification and Sentiment Analysis | by Abhijit Roy | Towards Data Science

Motivation: Text Classification and sentiment analysis is a very common machine learning problem and is used in a lot of activities like product predictions, movie recommendations, and several others. Currently, for every machine learner new to this field, like myself, exploring this domain has become very important. After exploring the topic, I felt, if I share my experience through an article, it may help some people trying to explore this field. So, I will try to build the whole thing from a basic level, so this article may look a little long, but it has some parts you can skip if you want.
Text classification problems like sentimental analysis can be achieved in a number of ways using a number of algorithms. These are majorly divided into two main categories:
A bag of Word model: In this case, all the sentences in our dataset are tokenized to form a bag of words that denotes our vocabulary. Now each individual sentence or sample in our dataset is represented by that bag of words vector. This vector is called the feature vector. For example, ‘It is a sunny day’, and ‘The Sun rises in east’ are two sentences. The bag of words would be all the words in both the sentences uniquely.The second method is based on a time series approach: Here each word is represented by an Individual vector. So, a sentence is represented as a vector of vectors.
A bag of Word model: In this case, all the sentences in our dataset are tokenized to form a bag of words that denotes our vocabulary. Now each individual sentence or sample in our dataset is represented by that bag of words vector. This vector is called the feature vector. For example, ‘It is a sunny day’, and ‘The Sun rises in east’ are two sentences. The bag of words would be all the words in both the sentences uniquely.
The second method is based on a time series approach: Here each word is represented by an Individual vector. So, a sentence is represented as a vector of vectors.
This is the very basic difference between the two approaches. Now, let’s see in detail.
For the purpose, we will use, the IMDB review dataset. It has 50,000 reviews and their corresponding sentiments marked as “Positive” and “Negative”. So, first, we need to check and clean our dataset.
from bs4 import BeautifulSoupdef strip_html(text):  soup = BeautifulSoup(text, "html.parser")  return soup.get_text()import redef remove_between_square_brackets(text):  return re.sub('\[[^]]*\]', '', text)def denoise_text(text):  text = strip_html(text)  text = remove_between_square_brackets(text)  return textdef remove_special_characters(text, remove_digits=True):  pattern=r'[^a-zA-z0-9\s]'  text=re.sub(pattern,'',text)   return textdef Convert_to_bin(text, remove_digits=True):  if text=='positive':   text= 1     else:   text=0  return text
These are the preprocessing required to clean the reviews part and the Convert_to_binary is used to convert “Positive” sentiment to 1 and “Negative” to 0. So, after preprocessing we obtain:
If we count plot our target class division.
We have 25k positive sentiments and 25k negative sentiments. So, let’s split up the data.
X=df['review'].valuesY=df['sentiment'].valuesX_train, X_test, Y_train, Y_test= train_test_split(X,Y, test_size=0.3)
Vectorizers: Bag of words
In this method, we create a single feature vector using all the words in the vocabulary, that we obtain from tokenizing the sentences in the train sets. This feature vector is used to represent all the reviews in our set. Each word is basically regarded as a feature. So, the number of features is equal to the number of unique words in the vocabulary. Now, each sentence or review is regarded as a sample or record. If the word is present in that sample it has some values for the word which is regarded as the feature, and if the word is not present it is zero. So, if ‘It is a sunny day’, and ‘The Sun rises in east’ are two sentences. and their bag will have 10 words which will be the feature vector size. Say vector is: [‘It’, ‘is’, ‘a’, ‘sunny’, ‘day’, ‘The’, ‘Sun’, ‘rises’, ‘in’, ‘east’]. So, the first sentence is represented by [a,b,c,d,e,0,0,0,0,0], a,b,c,d,e are values depending on the scheme we use
So, each sample has the same feature set size which is equal to the size of the vocabulary. Now, the vocabulary is basically made of the words in the train set. All the samples of the train and test set are transformed using this vocabulary only. So, there may be some words in the test samples which are not present in the vocabulary, they are ignored.
They form very sparse matrices or feature sets. Similar to a normal classification problem, the words become features of the record and the corresponding tag becomes the target value. If we consider as a dataset, the samples or reviews will be the rows or records, the feature set of each record, or the feature columns corresponding to each record will be equal to the size of the vocabulary, where each word will be a feature. So, it is actually like a common classification problem with the number of features being equal to the distinct tokens in the training set.
This can be done in two ways:
Count Vectorizer: Here the count of a word in a particular sample or review. The count of that word becomes the value of the corresponding word feature. If a word in the vocab does not appear in the sample its value is 0. So basically if there are 10 sentences in the train set, Those 10 sentences are tokenized to create the bag. Say, the created bag length is 100 words. So, we will form a vector of length 100. For a particular sample, if there are 15 words, where there is one word appearing three times in that sample, the value corresponding to that word becomes 3 for that sample and so on.TF-IDF Vectorizer: It is a better approach. It calculates two things term frequency and inverse document frequency. Term frequency= No. of times the word appears in the sample. and IDF = log ( number of times the word appears in the sample/number of times the word appears in the whole document). This helps to note some differences like the word “The” appears with the same freq in almost all sentences while special words carrying significance like “good” don’t. So, these TF and IDF values of each word for a specific sample are multiplied to obtain the feature vectors for that sample.
Count Vectorizer: Here the count of a word in a particular sample or review. The count of that word becomes the value of the corresponding word feature. If a word in the vocab does not appear in the sample its value is 0. So basically if there are 10 sentences in the train set, Those 10 sentences are tokenized to create the bag. Say, the created bag length is 100 words. So, we will form a vector of length 100. For a particular sample, if there are 15 words, where there is one word appearing three times in that sample, the value corresponding to that word becomes 3 for that sample and so on.
TF-IDF Vectorizer: It is a better approach. It calculates two things term frequency and inverse document frequency. Term frequency= No. of times the word appears in the sample. and IDF = log ( number of times the word appears in the sample/number of times the word appears in the whole document). This helps to note some differences like the word “The” appears with the same freq in almost all sentences while special words carrying significance like “good” don’t. So, these TF and IDF values of each word for a specific sample are multiplied to obtain the feature vectors for that sample.
We will be using scikit-learn’s feature extraction libraries here. But one thing to notice here is, this can also be done using TensorFlow's Tokenizer function. It can be done in mainly three ways using tokenizer:
Binary: X = tokenizer.sequences_to_matrix(x, mode=’binary’): In this case, the value of a word feature is 1 if the word is present in the sample else zero.Count: X = tokenizer.sequences_to_matrix(x, mode=’count’) : In this case, the value is the number of times a word appear in the sentence.TF-IDF: X = tokenizer.sequences_to_matrix(x, mode=’tfidf’): In this, we consider the TF of the word in the sample and IDF of the word in the sample with respect to the occurrence of the word in the whole document.
Binary: X = tokenizer.sequences_to_matrix(x, mode=’binary’): In this case, the value of a word feature is 1 if the word is present in the sample else zero.
Count: X = tokenizer.sequences_to_matrix(x, mode=’count’) : In this case, the value is the number of times a word appear in the sentence.
TF-IDF: X = tokenizer.sequences_to_matrix(x, mode=’tfidf’): In this, we consider the TF of the word in the sample and IDF of the word in the sample with respect to the occurrence of the word in the whole document.
Count Vectorizer: Code Implementation
from sklearn.feature_extraction.text import CountVectorizervec = CountVectorizer()vec.fit(X_train)x_train=vec.transform(X_train)x_test=vec.transform(X_test)
Now, if we notice, the vector is fit only to X_train. Here is where the vocabulary is formed. So vocabulary contains only the words in the train set. Then we transform on both train and test set.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Densefrom tensorflow.keras.optimizers import Adammodel = Sequential()model.add(Dense(16, input_dim=x_train.shape[1], activation='relu'))model.add(Dense(16, activation='relu'))model.add(Dense(1, activation='sigmoid'))model.compile(loss='binary_crossentropy',optimizer='Adam',metrics=['accuracy'])
This is our model for the count vectorization approach. We can see that the input dimension is of size equal to the number of columns for each sample which is equal to the number of words in our vocabulary. The output is 1 i.e, sentiment Positive or Negative.
The model gives an accuracy of 87.3%
TF-IDF Vectorizer: Code Implementation
from sklearn.feature_extraction.text import TfidfVectorizervec = TfidfVectorizer()vec.fit(X_train)x_train=vec.transform(X_train)x_test=vec.transform(X_test)
Now, here we call the TF-IDF vectorizer and the data is transformed. Here we use a logistic regression model.
from sklearn.linear_model import LogisticRegressionlr = LogisticRegression()lr.fit(x_train, Y_train)
This model gives an accuracy of 89.4%
Word Embedding: Time series approach
In this method, the words are individually represented as a vector. In the case of the bag of words, all of the words in the vocabulary made up a vector. Say, there are 100 words in a vocabulary, so, a specific word will be represented by a vector of size 100 where the index corresponding to that word will be equal to 1, and others will be 0.
So, Each sample having a different number of words will basically have a different number of vectors, as each word is equal to a vector. Now, to feed a model we will need to have the same dimension for each sample, and as a result, padding is needed to make the number of words in each sample equal to each other.
Basically in the bag of words or vectorizer approach, if we have 100 words in our total vocabulary, and a sample with 10 words and a sample with 15 words, after vectorization both the sample sizes would be an array of 100 words, but here for the 10 words it will be a (10 x 100) i.e, 100 length vector for each of the 10 words and similarly for 15th one size will be (15 x 100). So, we need to find the longest sample and pad all others up to match the size.
We can do this in some ways:
One-Hot encoding: It is just taking the size of the vocabulary and making an array of that size with 0’s at all indices and 1 at only the index of the word, as we have discussed above. But these things provide us with very little information and create a very sparse matrix.
The Second Choice is the word embeddings.
The one-hot encoder is a pretty hard-coded approach. It is of a very high dimension and sparse with a very low amount of data. Embedding is a way to create a dense vector representation. It is of a lower dimension and helps to capture much more information. It more like captures the relationships and similarities between words using how they appear close to each other. For example, the king, queen, men, and women will have some relations.
Say, we are having 10k words are being embedded in a 300-dimensional embedding space. To do this, we declare the number of nodes in the embedding layer=300. Now, each word of the 10k words enters the embedding layer. Each of the words will be placed in a 300-dimensional plane based on their similarities with one another which is decided by several factors, like the order in which the words occur. Now, being placed in 300 Dimensional planes the words will have a 300 length tuple to represent it which are actually the coordinates of the point on the 300-dimensional plane. So, this 300-dimensional tuple becomes the new feature set or representing a vector for the word.
So, the vector for the word decreased from 10k to 300. The tuples serve as feature vectors between two words and the cosine angle between the vectors represents the similarity between the two words.
We can do this in two ways:
Using our own embeddingsUsing pre-trained embeddings
Using our own embeddings
Using pre-trained embeddings
Making our own embedding
Now, for the embedding, we need to send each sample through an embedding layer first then move to make them dense using embedding. These embedding layers see how the words are used, i.e, it tries to see if two words always occur together or are used in contrast. After judging all these factors the layer places the word in a position one the n-dimensional embedding space.
Using pre-trained embedded matrices
We can use pre-trained word embeddings like word2vec by google and GloveText by Standford. They are trained on huge corpora with billions of examples and words. Now, they have billions of words we have only say, a 10k so, training our model with a billion words will be very inefficient. We need to just select out our required word’s embeddings from their pre-trained embeddings.
Now, How are these embeddings found?
For google’s word2vec implementations, there are two ways:
Continous bag of wordsSkip-Gram.
Continous bag of words
Skip-Gram.
Both of these algorithms actually use a Neural Network with a single hidden layer to generate the embedding.
For CBOW, the context of the words, i.e, the words before, and after the required words are fed to the neural network, and the model is needed to predict the word.
For the Skip-Gram, the words are given and the model has to predict the context words.
In both cases, the feature vectors or encoded vectors of the words are fed to the input. The output has a softmax layer with a number of nodes equal to the vocabulary size, which gives the percentage of prediction for each word, i.e, it tells what is the probability that the required word is the word, the node in the softmax layer is representing. Though we don’t use the output layer actually.
We go for the weight matrix produced in the hidden layer. The number of nodes in the hidden layer is equal to the embedding dimension. So, say if there are 10k words in vocabulary and 300 nodes in the hidden layer, each node in the hidden layer will have an array of weights of the dimension of 10k for each word after training.
Because the neural network units work on
y=f(w1x1+w2x2+..........wnxn)
Here x1, x2...... xn are the words and so n= number of words in vocabulary=10k for our case.
So for 10k x’s, there will be 10k w’s. Now for 1 node, there is a 10k length-weight matrix. For 300 combined we have a matrix of 300 x 10k weights. Now, if we concatenate, we will have 300 rows and 10k columns. Let’s transpose the matrix. We will get 300 columns and 10k rows. Each row represents a word, and the 300 column values represent a 300 length-weight vector for that word.
This weight vector is the obtained embedding of length 300.
Let’s look at implementations:
Training Own Embedding
from keras.preprocessing.text import Tokenizertokenizer = Tokenizer(num_words=10000)tokenizer.fit_on_texts(X_train)
Num_words indicates the size of the vocabulary
x_train = tokenizer.texts_to_sequences(X_train)x_test = tokenizer.texts_to_sequences(X_test)
1) tokenize.fit_on_text() →> Creates the vocabulary index based on word frequency. For example, if you had the phrase “My dog is different from your dog, my dog is prettier”, word_index[“dog”] = 0, word_index[“is”] = 1 (‘dog’ appears 3 times, ‘is’ appears 2 times)
2) tokenize.text_to_sequence() →> Transforms each text into a sequence of integers. Basically, if you had a sentence, it would assign an integer to each word from your sentence. You can access tokenizer.word_index() (returns a dictionary) to verify the assigned integer to your word.
vocab = len(tokenizer.word_index) + 1from keras.preprocessing.sequence import pad_sequencesmaxlen = 100x_train = pad_sequences(x_train, padding='post', maxlen=maxlen)x_test = pad_sequences(x_test, padding='post', maxlen=maxlen)
We are padding all sentences to a length of max length 100.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1Dfrom tensorflow.keras.optimizers import Adamemb_dim=100model= Sequential()model.add(Embedding(input_dim=vocab, output_dim=emb_dim, input_length=maxlen))model.add(MaxPool1D())model.add(Dense(16,activation="relu"))model.add(Dense(16,activation="relu"))model.add(Dense(1, activation='sigmoid'))model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])
Max pool layer is used to pick out the best-represented features to decrease sparsity.
This model gives an accuracy of 77.4%
Using Pre-trained Embeddings
We will be using Standford's Glove embedding which is trained over 6Billion words.
It has four files each with a different embedding space, we will be using the 50d one, which is a 50-Dimensional Embedding space.
['glove.6B.100d.txt',  'glove.6B.200d.txt',  'glove.6B.50d.txt',  'glove.6B.300d.txt']
Implementations:
import numpy as npfrom tensorflow.keras.preprocessing.text import Tokenizertokenizer = Tokenizer(num_words=10000)tokenizer.fit_on_texts(X_train)x_train = tokenizer.texts_to_sequences(X_train)x_test = tokenizer.texts_to_sequences(X_test)from keras.preprocessing.sequence import pad_sequencesmaxlen=100x_train = pad_sequences(x_train, padding='post', maxlen=maxlen)x_test = pad_sequences(x_test, padding='post', maxlen=maxlen)
So, let’s see how to extract the embedding we require from the given embedding file.
emb_dim=50vocab=len(tokenizer.word_index)+1emb_mat= np.zeros((vocab,emb_dim))#Initializing a zero matrix for each word, they will be compared to have their final embeddingwith open(file_path+'glove.6B.50d.txt') as f:  for line in f:    word, *emb = line.split()     if word in tokenizer.word_index:    ind=tokenizer.word_index[word]    emb_mat[ind]=np.array(emb,dtype="float32")[:emb_dim]
This is an extractor for the task, so we have the embeddings and the words in a line. So, we just compare the words to pick out the indices in our dataset. Take the vectors and place them in the embedding matrix at an index corresponding to the index of the word in our dataset.
We have used an *emb because the embedding matrix is variant in size.
So, we here have a feature set with a vocabulary of 10k words and each word represented by a 50 length tuple embedding which we obtained from the Glove embedding.
Next, we build our model.
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1Dfrom tensorflow.keras.optimizers import Adamemb_dim=50maxlen=100model= Sequential()model.add(Embedding(input_dim=vocab, output_dim=emb_dim,weights=[emb_mat], input_length=maxlen,trainable=False))model.add(MaxPool1D())model.add(Dense(16,activation="relu"))model.add(Dense(16,activation="relu"))model.add(Dense(1, activation='sigmoid'))model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])
This model gives an accuracy of 67% probably due to the decreased embedding size. The larger the embedding size more the information contained.
Now, how these approaches are beneficial over to the bag of words model? As we can see these Bag of Words models just saw how a word behaves in the document, i.e, what can we tell about the frequency of occurrence of the word or any pattern in which the word occurs? While these approaches also take into consideration the relationship between two words using the embeddings.
In mathematics (in particular, functional analysis) convolution is a mathematical operation on two functions (f and g) that produces a third function expressing how the shape of one is modified by the other. The term convolution refers to both the result function and to the process of computing it.
So, Convolutional is best for extracting special features and behavior of feature values from the 2D pixels of images. Convolutional layers have a set of kernels which helps to extract several important features from the data samples. Now here, in case of text classifications, our feature matrices are 1Dimensional. So, here Conv1D is used. Basically it moves as a sliding window of size decided by the user. We have chosen 5.
Now, initially after embedding, we get 100 Dimensional embeddings. Next using 1D convolutions we try to make our feature set smaller and let the feature set discover the best features relations for the classification. The max-pooling layer also helps to pick the features or words which have the best performance.
The convolutional layer is always used after an embedding layer after it provides its embedded feature vectors.
Implementations:
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1D,Conv1Dfrom tensorflow.keras.optimizers import Adamemb_dim=100model= Sequential()model.add(Embedding(input_dim=vocab, output_dim=emb_dim,input_length=maxlen))model.add(Conv1D(64, 5, activation='relu'))model.add(MaxPool1D(5))model.add(Conv1D(128, 5, activation='relu'))model.add(MaxPool1D(5))model.add(Dense(16,activation="relu"))model.add(Dense(1, activation='sigmoid'))model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])
Model Summary
Here we have trained our own embedding matrix of dimension 100. We have used two convolutional layers with 64 and 128 kernel filters respectively. We have kept our sliding window =5.
This model has given a test accuracy of 77%
Why Recurrent Neural Networks?
Until now we have tried to extract some features from all the words in a sample at a time. So, all of them are non-temporal approaches. Now, let’s see how a person will judge a sentiment. He/she will not only consider what were the words used, but humans will also consider how they are used, that is, in what context, and what are the preceding and succeeding words? So, until now we have focused on what were the words used only, so, now let’s look at the other part of the story.
So, for this part, we need a Recurrent neural network to give a memory to our models. If we think about telling something about someone's statements, we will generally listen to the whole statement word by word and then make a comment. This is what the Recurrent Neural networks will accomplish. It will look at each word in a temporal manner one by one and try to correlate to the context using the embedded feature vector of the word.
We know RNN suffers from the vanishing and exploding gradient problem we will be using LSTM.
LSTM operates on two things a hidden state that is sent from a previous timestamp and a cell state that actually maintains the weight neutralizing the vanishing gradient effect.
The LSTM layer basically has 4 components: A Forget gate, An input gate, a cell state, and an output gate.
Now, let’s talk a bit about the working and dataflow in an LSTM, as I think this will help to show how the feature vectors are actually formed and what it looks like.
These are the set of equations LSTM operates upon.
LSTM provides a feature set on the last timestamp for the dense layer, to use the feature set to produce results. We can see the above equations are the equations for the Gates of LSTM. Here, each gate acts as a neural network individually. So, they have their individual weight matrices that are optimized when the recurrent network model is trained. Using these weight matrices only the gates learn their tasks, like which data to forget and what part of the data is needed to be updated to the cell state. So, the gates optimize their weight matrices and decide the operations according to it.
Now, let's see the use.
Say we have a 100-dimensional vector space. a batch size of 16, each sample length = 10. and the number of nodes in each layer= 64.
INPUT SIZE = batch_size * Embedding so, here it is 16 x 100 matrix = x(t)
The timestamp 0 that is the first word of every sample or record enters.
The x0 represents the first word of the samples, x1 represents second, and so on. So, each time 1 word from 16 samples and each word is represented by a 100 length vector. So, the size of each input is (16 x 100).
PREVIOUS HIDDEN STATE (0 vector for tiemstamp 0) = Batch size x Hidden Units So, Here it is 16 x 64 matrix.= h(t-1)
After concatenation, the matrix formed h(t-1)= 16 x 64 and x(t)= 16 x 100
So the h(t-1) + x(t) matrix is sent to all gates f(t) , u(t) and o(t) .
Forget Gate
First, the Forget Gate Weight matrix W{hf} of the hidden state is of dimension 64 x 64 because in the hidden state for each of the 16 words of timestamp (t-1) there were 64 values from each of the 64 nodes of the RNN.
So, actually our matrix from the hidden state with shape (16 x 64):16 rows which are records, and for each record there are 64 columns or 64 features.
We know for each node,
y=w1x1+w2x2+........wnxn
the equation holds.
where the x’s are the features or the column values. So, there must a maintained array of 64 weights, one corresponding to each x, for each node or unit of the network. Now there are 64 such units so a total of (64 x 64) matrix.
Again, now for the input vector of shape (16 x 100), there are 16 rows or records for each of the 100 columns or 100 features. so, the weight matrix of one hidden unit must have 100 values. A total of 64 units are there. So, the dimension of the weight matrix W{xf} is. (100 x 64)
So at forget gate
according to equation 1.
F{t}= sigmoid((16 x 100) x (100 x 64) + (16 x 100) x (100 x 64))
F{t}=sigmoid (16 x 64) vector
Sigmoid gives a value between 0 and 1. If the value is close to 0 the value is forgotten else added to the cell state after passing through F{t}.
Now, the cell state is also of the same dimension (16 x 64) as it is also having the weights of the 16 sample word’s by 64 nodes So, they can easily be added.
Update Gate
Next, is the input or update gate it decides what part of the data should enter, which means the actual update function of the Recurrent Neural Network. It decides whether the cell state should be updated.
These gate’s weight matrices are also the same as the forget gate’s matrices with (64 x 64) values in the last hidden layer and (100 x 64) values for the input.
W{hu} is of dimension (64x 64) and W{xu} is of dimension (100 x 64). The Weight matrix with ‘h’ in subscript are multiplied with the h(t-1) portion and correspondingly weight matrices with ‘x’ in subscript are multiplied with the x(t) portion of the concatenated {x(t) +h(t-1)} vector obtained by adding the previously hidden layer with current input.
Every weight matrix with h has dimension (64 x 64) and Every weight matrix with x has dimension (100 x 64).
So, Input gate also gives sigmoid(16 x 64) as a result
According to equation 2,
U{t}= sigmoid((16 x 100) x (100 x 64) + (16 x 100) x (100 x 64))
U{t}=sigmoid (16 x 64) vector
One thing to notice here is there is a tanh layer also. The tanh is here to squeeze the value between 1 to -1 to deal with the exploding and vanishing gradient. So, it basically works like and regularized value that represents the value of the temporary cell state on that timestep. The sigmoid is the switch.
According to equation 3, the temporary cell state is calculated.
C1{t}= tanh((16 x 100) x (100 x 64) + (16 x 100) x (100 x 64))
C1{t}=tanh (16 x 64) vector
The C1{t} and the U{t} vector matrices are of the same dimensions. So, dot product can be applied.
So, after that, the obtained vectors are just multiplied to obtain 1 result.
One thing to notice about this is, though the weight matrices are of the same dimensions, but, they are not the same. They belong to different gates and their values and optimizations are all different.
Cell State
Now in the next step, the cell step is updated
It is basically a simple sum.
The new c or cell state is formed by removing the unwanted information from the last step + accomplishments of the current time step. as shown according to equation 5.
Output Gate
Next, comes the output gate. This decides what should be the next steps hidden layer be.
For this, the new cell state is passed through a tanh gate and the h(t-1) + x(t) is passed through another sigmoid. Both of the results are multiplied. That’s the next.
According to equation 4, the output gate which decides the next hidden layer.
O{t}= sigmoid((16 x 100) x (100 x 64) + (16 x 100) x (100 x 64))
O{t}=sigmoid (16 x 64) vector
So here we can see the dimension of O(t) and the h(t-1) matches.
Now, these weights get updated at every timestep with every word, and after the 10th word or timestamp, the final timestamp in our case the model has gone through all the words in the samples, so we get a matrix of size 16 x 64, which is basically the weight values of the 64 internal nodes corresponding to each sample. But, what we don’t see are the weight matrices of the gates which are also optimized. These 64 values in a row basically represent the weights of an individual sample in the batch produced by the 64 nodes, one by each .
For all the samples we obtain a value. These values act as a feature set for the dense layers to perform their operations.
Implementations:
from tensorflow.keras.models import Sequentialfrom tensorflow.keras.layers import Embedding,Dense, Activation, MaxPool1D, LSTMfrom tensorflow.keras.optimizers import Adamemb_dim=50maxlen=100model= Sequential()model.add(Embedding(input_dim=vocab, output_dim=emb_dim,weights=[emb_mat], input_length=maxlen,trainable=False))model.add(MaxPool1D())model.add(LSTM(64, return_sequences = False))model.add(Dense(16,activation="relu"))model.add(Dense(1, activation='sigmoid'))model.compile(optimizer='Adam',loss='binary_crossentropy',metrics=['accuracy'])
Here, we have used a pre-trained word embedding. We have used 1 LSTM layer with 64 hidden unit nodes.
This model provides an accuracy of 78%
It is a hybrid model formed by combining an LSTM layer followed by a CNN layer.
The LSTM layer is generating a new encoding for the original input. The output of the LSTM layer is then fed into a convolution layer which we expect will extract local features. Finally, the convolution layer’s output will be pooled to a smaller dimension and ultimately outputted as either a positive or negative label.
We have talked about a number of approaches that can be used for text classification problems like the sentimental analysis.
I hope this helps.
The Github link is here.