Sentiment Analysis using LSTM Step by Step Tutorial | Deep Learning | Towards Data Science

I think this result from google dictionary gives a very succinct definition. I don’t have to re-emphasize how important sentiment analysis has become. So, here we will build a classifier on IMDB movie dataset using a Deep Learning technique called RNN.
I’m outlining a step-by-step process for how Recurrent Neural Networks (RNN) can be implemented using Long Short Term Memory (LSTM) architecture:
Load in and visualize the dataData Processing — convert to lower caseData Processing — Remove punctuationData Processing — Create list of reviewsTokenize — Create Vocab to Int mapping dictionaryTokenize — Encode the wordsTokenize — Encode the labelsAnalyze Reviews LengthRemoving Outliers — Getting rid of extremely long or short reviewsPadding / Truncating the remaining dataTraining, Validation, Test Dataset SplitDataloaders and BatchingDefine the LSTM Network ArchitectureDefine the Model ClassTraining the NetworkTesting (on Test data and User- generated data)
Load in and visualize the data
Data Processing — convert to lower case
Data Processing — Remove punctuation
Data Processing — Create list of reviews
Tokenize — Create Vocab to Int mapping dictionary
Tokenize — Encode the words
Tokenize — Encode the labels
Analyze Reviews Length
Removing Outliers — Getting rid of extremely long or short reviews
Padding / Truncating the remaining data
Training, Validation, Test Dataset Split
Dataloaders and Batching
Define the LSTM Network Architecture
Define the Model Class
Training the Network
Testing (on Test data and User- generated data)
1) Load in and visualize the data
We are using IMDB movies review dataset. If it is stored in your machine in a txt file then we just load it in
# read data from text fileswith open(‘data/reviews.txt’, ‘r’) as f: reviews = f.read()with open(‘data/labels.txt’, ‘r’) as f: labels = f.read()print(reviews[:50])print()print(labels[:26])--- Output ---bromwell high is a cartoon comedy . it ran at the same time as some other programs about school life  such as  teachers  . my   yearspositivenegativepositive
2) Data Processing — convert to lower case
reviews = reviews.lower()
3) Data Processing — remove punctuation
from string import punctuationprint(punctuation)--- Output ---!"#$%&'()*+,-./:;<=>?@[\]^_`{|}~
We saw all the punctuation symbols predefined in python. To get rid of all these punctuation we will simply use
all_text = ''.join([c for c in reviews if c not in punctuation])
4) Data Processing — create list of reviews
We have got all the strings in one huge string. Now we will separate out individual reviews and store them as individual list elements. Like, [review_1, review_2, review_3....... review_n]
reviews_split = all_text.split(‘\n’)print ('Number of reviews :', len(reviews_split))
Number of reviews : 25001
5) Tokenize — Create Vocab to Int mapping dictionary
In most of the NLP tasks, you will create an index mapping dictionary in such a way that your frequently occurring words are assigned lower indexes. One of the most common way of doing this is to use Counter method from Collections library.
from collections import Counterall_text2 = ' '.join(reviews_split)# create a list of wordswords = all_text2.split()# Count all the words using Counter Methodcount_words = Counter(words)total_words = len(words)sorted_words = count_words.most_common(total_words)
Let’s have a look at these objects we have created
print (count_words)--- Output ---Counter({'the': 336713, 'and': 164107, 'a': 163009, 'of': 145864
In order to create a vocab to int mapping dictionary, you would simply do this
vocab_to_int = {w:i for i, (w,c) in enumerate(sorted_words)}
There is a small trick here, in this mapping index will start from 0 i.e. mapping of ‘the’ will be 0. But later on we are going to do padding for shorter reviews and conventional choice for padding is 0. So we need to start this indexing from 1
vocab_to_int = {w:i+1 for i, (w,c) in enumerate(sorted_words)}
Let’s have a look at this mapping dictionary. We can see that mapping for ‘the’ is 1 now
print (vocab_to_int)--- Output ---{'the': 1, 'and': 2, 'a': 3, 'of': 4,
6) Tokenize — Encode the words
So far we have created a) list of reviews and b) index mapping dictionary using vocab from all our reviews. All this was to create an encoding of reviews (replace words in our reviews by integers)
reviews_int = []for review in reviews_split:    r = [vocab_to_int[w] for w in review.split()]    reviews_int.append(r)print (reviews_int[0:3])--- Output ---[[21025, 308, 6, 3, 1050, 207, 8, 2138, 32, 1, 171, 57, 15, 49, 81, 5785, 44, 382, 110, 140, 15, .....], [5194, 60, 154, 9, 1, 4975, 5852, 475, 71, 5, 260, 12, 21025, 308, 13, 1978, 6, 74, 2395, 5, 613, 73, 6, 5194, 1, 24103, 5, ....], [1983, 10166, 1, 5786, 1499, 36, 51, 66, 204, 145, 67, 1199, 5194.....]]
Note: what we have created now is a list of lists. Each individual review is a list of integer values and all of them are stored in one huge list
7) Tokenize — Encode the labels
This is simple because we only have 2 output labels. So, we will just label ‘positive’ as 1 and ‘negative’ as 0
encoded_labels = [1 if label =='positive' else 0 for label in labels_split]encoded_labels = np.array(encoded_labels)
8) Analyze Reviews Length
import pandas as pdimport matplotlib.pyplot as plt%matplotlib inlinereviews_len = [len(x) for x in reviews_int]pd.Series(reviews_len).hist()plt.show()pd.Series(reviews_len).describe()
Observations : a) Mean review length = 240 b) Some reviews are of 0 length. Keeping this review won’t make any sense for our analysis c) Most of the reviews less than 500 words or more d) There are quite a few reviews that are extremely long, we can manually investigate them to check whether we need to include or exclude them from our analysis
9) Removing Outliers — Getting rid of extremely long or short reviews
reviews_int = [ reviews_int[i] for i, l in enumerate(reviews_len) if l>0 ]encoded_labels = [ encoded_labels[i] for i, l in enumerate(reviews_len) if l> 0 ]
10) Padding / Truncating the remaining data
To deal with both short and long reviews, we will pad or truncate all our reviews to a specific length. We define this length by Sequence Length. This sequence length is same as number of time steps for LSTM layer.
For reviews shorter than seq_length, we will pad with 0s. For reviews longer than seq_length we will truncate them to the first seq_length words.
def pad_features(reviews_int, seq_length):    ''' Return features of review_ints, where each review is padded with 0's or truncated to the input seq_length.    '''    features = np.zeros((len(reviews_int), seq_length), dtype = int)        for i, review in enumerate(reviews_int):        review_len = len(review)                if review_len <= seq_length:            zeroes = list(np.zeros(seq_length-review_len))            new = zeroes+review        elif review_len > seq_length:            new = review[0:seq_length]                features[i,:] = np.array(new)        return features
Note: We are creating/maintaining a 2D array structure as we created for reviews_int . Output will look like this
print (features[:10,:])
11) Training, Validation, Test Dataset Split
Once we have got our data in nice shape, we will split it into training, validation and test sets
train= 80% | valid = 10% | test = 10%
split_frac = 0.8train_x = features[0:int(split_frac*len_feat)]train_y = encoded_labels[0:int(split_frac*len_feat)]remaining_x = features[int(split_frac*len_feat):]remaining_y = encoded_labels[int(split_frac*len_feat):]valid_x = remaining_x[0:int(len(remaining_x)*0.5)]valid_y = remaining_y[0:int(len(remaining_y)*0.5)]test_x = remaining_x[int(len(remaining_x)*0.5):]test_y = remaining_y[int(len(remaining_y)*0.5):]
12) Dataloaders and Batching
After creating our training, test and validation data. Next step is to create dataloaders for this data. We can use generator function for batching our data into batches instead we will use a TensorDataset. This is one of a very useful utility in PyTorch for using our data with DataLoaders with exact same ease as of torchvision datasets
import torchfrom torch.utils.data import DataLoader, TensorDataset# create Tensor datasetstrain_data = TensorDataset(torch.from_numpy(train_x), torch.from_numpy(train_y))valid_data = TensorDataset(torch.from_numpy(valid_x), torch.from_numpy(valid_y))test_data = TensorDataset(torch.from_numpy(test_x), torch.from_numpy(test_y))# dataloadersbatch_size = 50# make sure to SHUFFLE your datatrain_loader = DataLoader(train_data, shuffle=True, batch_size=batch_size)valid_loader = DataLoader(valid_data, shuffle=True, batch_size=batch_size)test_loader = DataLoader(test_data, shuffle=True, batch_size=batch_size)
In order to obtain one batch of training data for visualization purpose we will create a data iterator
# obtain one batch of training datadataiter = iter(train_loader)sample_x, sample_y = dataiter.next()print('Sample input size: ', sample_x.size()) # batch_size, seq_lengthprint('Sample input: \n', sample_x)print()print('Sample label size: ', sample_y.size()) # batch_sizeprint('Sample label: \n', sample_y)
Here, 50 is the batch size and 200 is the sequence length that we have defined. Now our data prep step is complete and next we will look at the LSTM network architecture for start building our model
13) Define the LSTM Network Architecture
The layers are as follows:
0. Tokenize : This is not a layer for LSTM network but a mandatory step of converting our words into tokens (integers)
Embedding Layer: that converts our word tokens (integers) into embedding of specific sizeLSTM Layer: defined by hidden state dims and number of layersFully Connected Layer: that maps output of LSTM layer to a desired output sizeSigmoid Activation Layer: that turns all output values in a value between 0 and 1Output: Sigmoid output from the last timestep is considered as the final output of this network
Embedding Layer: that converts our word tokens (integers) into embedding of specific size
LSTM Layer: defined by hidden state dims and number of layers
Fully Connected Layer: that maps output of LSTM layer to a desired output size
Sigmoid Activation Layer: that turns all output values in a value between 0 and 1
Output: Sigmoid output from the last timestep is considered as the final output of this network
Note: If you want to understand more about these LSTM layers and get a microscopic view of things. Read this one —
towardsdatascience.com
14) Define the Model Class
15) Training the Network
Instantiate the network
# Instantiate the model w/ hyperparamsvocab_size = len(vocab_to_int)+1 # +1 for the 0 paddingoutput_size = 1embedding_dim = 400hidden_dim = 256n_layers = 2net = SentimentLSTM(vocab_size, output_size, embedding_dim, hidden_dim, n_layers)print(net)SentimentLSTM(  (embedding): Embedding(74073, 400)  (lstm): LSTM(400, 256, num_layers=2, batch_first=True, dropout=0.5)  (dropout): Dropout(p=0.3)  (fc): Linear(in_features=256, out_features=1, bias=True)  (sig): Sigmoid())
Training Loop
Most of the code in training loop is pretty standard Deep Learning training code that you might see often in all the implementations that’s using PyTorch framework.
16) Testing
On Test Data
On User-generated Data
First, we will define a tokenize function that will take care of pre-processing steps and then we will create a predict function that will give us the final output after parsing the user provided review.
test_review = 'This movie had the best acting and the dialogue was so good. I loved it.'seq_length=200 # good to use the length that was trained onpredict(net, test_review_neg, seq_length)
Positive review detected
If you enjoyed this, follow me on medium for more.
Your claps are a huge encouragement and help in writing more and writing better.
Interested in collaborating? Let’s connect on Linkedin.
Please feel free to write your thoughts/suggestions/feedback.
Update: Another article to give you a microscopic view of what happens within the layers.