Finding the hidden sentiments using Deep Learning in Pytorch
By: Dipika Baad
Tags: Machine Learning, Sentiment Analysis, Neural Networks, Rnn, Naturallanguageprocessing

The aim of this blog is to introduce you in a simple and easy way what is the role of RNNs(Recurrent Neural Networks) in natural language processing tasks and how RNNs can be used to be used for sentiment classification with it’s implementation for twitter data. Sentiment analysis has become integral part of so many upcoming applications these days. Almost all brands nowadays want to know the public’s overall attitude towards them from online comments, majority of people go through reviews before buying or booking anything online, financial industries are keeping an eye on the developments of the trends and be it the politics where hot topics or public opinions are analyzed. Hence, answering these questions is where sentiment analysis can be used certainly. So let’s dive into this interesting problem with RNNs!

Why should we use RNNs for this?

Order of the words in a sentence contributes a lot in predicting the sentiment of the sentence accurately. Well, why can’t we just use the simple bag of words and based on the sentiment of individual words predict what the whole paragraph is all about. Hmm… this is a no brainer, text gets its’ real meaning due to the sequence in which words are used. Not to forget tweets and conversations online are full of sarcasms and twisted usage of words which requires complex system to capture the real sentiment of those. Let’s have a look at a review done for hotel booking:

We had requested for two double rooms but surprise surprise they gave us two single rooms!

Do you see where I am going with this!! This sentence does not contain a lot of negative words to start of with so considering the connotation of the individual words for deriving the ultimate sentiment is out of the question. If I just flip few words this review would turn out to be positive ( from the point of view of gaining better rooms).

We had requested for two single rooms but surprise surprise they gave us two double room!

Who wouldn’t enjoy the double rooms right!? Certainly the hotel is being too generous with extra offer surprisingly or they were trying this surprise offers to attract more customers then capturing this review under positive sentiment would have helped them to get overview faster. Thus if we are not able to capture the sequence if the “single room” is coming before the word “but” or “double room” before the “but” the real sentiment will be hard to capture. This is just one example, you can easily imagine how the order of words plays key role in understanding the sentiment of the sentence. So if you want to make your sentiment analyzer smarter, the sequence of words needs a special attention.

The way to capture the sequence implies we need to have some sort of memory to remember the previous words and this is exactly the feature that RNNs provide over the normal feed-forward neural network. RNNs can remember the past hidden state which essentially helps to steer the final output.

RNNs Architecture

RNNs to be simply put are neural networks with loops in them. As you might have guessed, the loops are the ones used to feed the previous state for using the information from the past. Indeed facilitating the flow of information from previous state while you are going through the next input. The input will be provided with one word at a time so the input at time x_t will receive the h_t-1 the previous hidden state when x_t-1 was feeded to the network. As you can see from the figure below the W_hh is the additional weight matrix which will help to learn the weights for connecting the previous hidden state to the next. Essentially, now the output is not a mere function of the input but is the function of the (current input + previous state). Input is only used to change the state of the memory stored unlike the usual network where it has direct effect on the output.

RNN Unfolded Network with cycles

At each step the output h_t-1 generated from the previous input x_t-1 is fed to the processing and this is followed until the last element in the sequence. The equation for calculating the h_t at time step t is as follows:

h_t = f( (W_hh)(h_t-1) + (W_hx)(x_t))

Here the f is usually the tanh or ReLU function. The output y_t at time step t will be the probability distribution over the classes which can be derived from the softmax function.

y_t = softmax((W_yh)(h_t))

The final output from the last word in the sequence is the only one we care about for sentiment classification, but in case of problem of predicting the next word, outputs at each time step would have been considered.

During the training of the neural network, RNNs use the Backpropagation through time(BPTT). First you calculate loss based on the h_t and actual value and then use the BPTT. To train a neural network, stochastic gradient descent(SGD) via back propagation is often used to minimize the loss function. If you know how the back-propagation works, the same issue of vanishing grading with increasing number of layers applies to RNNs as well. Here you can imagine each time step to be a layer (only in our case we will have same weights) then the very first time step will get the least adjustments to its weights as gradients and thus it is as good as not remembering the first few words in the sentence. RNNs suffer from the vanishing gradient problem. This is explained quite well visually in this article which makes it real easy to understand this problem as well as the training process.

Now, that you have got some idea about the internal mechanisms of RNNs, let’s see how to implement in pytorch :)

Implementing in Pytorch

The dataset being used for experimenting the sentiment classification with RNNs is twitter data from Kaggle. The dataset contains the tweets about U.S. Airlines and has three sentiments about the tweets, this includes the tags:

Positive Neutral Negative

Showing the sentiment of the tweets posted by people for these airlines. Our goal here would be to predict the sentiment based on the small texts of tweets.

1. Loading and Pre-processing of the Data

First we load the necessary libraries. I am using the torch 1.0.0 . In the preprocessing part, for every sentence we need to split them into tokens first. I have used Porter Stemmer to stem the words as well. Removal of stopwords is done using nltk package. While reading the file, I am adding the tweet text in the tweets variable and collecting the sentiment class of that corresponding tweet in tweet_sent_class . sentiment_class is used to keep track of unique classes.( Here we know it’s three classes but for your future program this can be used to detect the classes automatically)

#Importing the necessary libraries

import csv

import torch

from nltk.corpus import stopwords

from nltk.stem import PorterStemmer

import torch.nn as nn

import torch.nn.functional as F

import torch.optim as optim from sklearn.metrics import accuracy_score

from sklearn.metrics import confusion_matrix # Variable for location of the tweets.csv file

INPUTFILE_PATH = "data/twitter-airline-sentiment/Tweets.csv" tweets = []

train_tweets =[]

test_tweets = []

sentiment_class = set()

tweet_sent_class= []

porter = PorterStemmer()

stop_words = set(stopwords.words('english')) def tokenizer(sentence):

tokens = sentence.split(" ")

tokens = [porter.stem(token.lower()) for token in tokens if not token.lower() in stop_words]

return tokens i = 0

with open(INPUTFILE_PATH, 'r') as csvfile:

tweetreader = csv.reader(csvfile, delimiter=',', quotechar='"')

for row in tweetreader: # For skipping the headerline

if i == 0:

i += 1

continue

# tweets will contain the tweet text

tweets.append(tokenizer(row[10]))

tweet_sent_class.append(row[1])

sentiment_class.add(row[1])

i += 1

2. Building the feature vector

Here, we build the vocabulary to map the unique index to the words for getting the input vector for the sentence. Each word would then be passed through the word embedding layer to obtain the word feature vector which will be the actual input to our RNN. In the following code, the function map_word_vocab will create the tensor for the ids from vocab for each word in the sentence. The map_class will encode the output for each sentence in the vector with index of the sentiment class. We set the EMBEDDING_DIM to 50 and HIDDEN_DIM to 10 which will be the size of the hidden layer in our RNN network. EMBEDDING_DIM is the size of the word embeddings that we want. Train and Test split is done with first 9000 documents for training and 5640 for testing.

class_dict = {}

for index, class_name in enumerate(sentiment_class):

class_dict[class_name] = index vocab = {}

vocab_index = 0

for tokens in tweets:

for key, token in enumerate(tokens):

#all_tokens.add(token)

if token not in vocab:

vocab[token] = vocab_index

vocab_index += 1 #train test split

train_tweets = tweets[:9000]

test_tweets = tweets[9000:] def map_word_vocab(sentence):

idxs = [vocab[w] for w in sentence]

return torch.tensor(idxs, dtype=torch.long) def map_class(sentiment):

return torch.tensor([class_dict[sentiment]], dtype=torch.long) def prepare_sequence(sentence):

# create the input feature vector

input = map_word_vocab(sentence)

return input EMBEDDING_DIM = 50

HIDDEN_DIM = 10

3. Defining RNN Class

Now we finally move on to our RNN class. There will be 4 layers in this:

Input layer Embedding layer RNN layer Output layer

Input is the sequence of words. Each word is encoded in terms of unique word. For each word, we get the word embedding of dimension 50 and then each of these words are passed sequentially to our RNN network of hidden layer size of 10. Output layer will predict the class it belongs to at the end so it is of the size equal to the number of classes (=3). We initialize the RNN class at the end along with the loss function negative log likelihood and optimizer SGD .

class RNN(nn.Module):

def __init__(self, input_size, hidden_size, vocab_size, output_size):

super(RNN, self).__init__() self.hidden_size = hidden_size

self.word_embeddings = nn.Embedding(vocab_size, input_size) self.i2h = nn.Linear(input_size + hidden_size, hidden_size)

self.i2o = nn.Linear(input_size + hidden_size, output_size)

self.softmax = nn.LogSoftmax(dim=1) def forward(self, word, hidden):

embeds = self.word_embeddings(word)

combined = torch.cat((embeds.view(1, -1), hidden), 1)

hidden = self.i2h(combined)

output = self.i2o(combined)

output = self.softmax(output)

return output, hidden def init_hidden(self):

return torch.zeros(1, self.hidden_size) # creating an instance of RNN

rnn = RNN(EMBEDDING_DIM, HIDDEN_DIM, len(vocab), len(sentiment_class))

# Setting the loss function and optimizer

loss_function = nn.NLLLoss()

optimizer = optim.SGD(rnn.parameters(), lr=0.001)

As you can see from the class definition, nn.embeddings is used to create the word embeddings, the layers between input and hidden as well as between input to output are simple linear layers. Output layer is softmax layer. You can have a quick look at the architecture of this from the pytorch tutorial of character level classification using RNN (Network diagram) which I had referred.

4. Training the Network

After this, we are ready to train our neural network. I ran this with 30 epochs and for each epoch we go through all the rows in the training dataset where first we get the feature vector and output vector. As it can be seen when going through each word, we get back the hidden state at each step. After the forward pass, we calculate the loss and do the backward propagation with SGD.

for epoch in range(30):

if epoch % 5 == 0:

print("Finnished epoch " + str(epoch / 30 * 100) + "%")

for i in range(len(train_tweets)):

sentence = train_tweets[i]

sent_class = tweet_sent_class[i] # Step 1. Remember that Pytorch accumulates gradients.

# We need to clear them out before each instance # Also, we need to clear out the hidden state of the LSTM,

# detaching it from its history on the last instance.

hidden = rnn.init_hidden()

rnn.zero_grad() # Step 2. Get our inputs ready for the network, that is, turn them into

# Tensors of word indices.

sentence_in = prepare_sequence(sentence)

target_class = map_class(sent_class)



# Step 3. Run our forward pass.

for i in range(len(sentence_in)):

class_scores, hidden = rnn(sentence_in[i], hidden)



# Step 4. Compute the loss, gradients, and update the parameters by

# calling optimizer.step()

loss = loss_function(class_scores, target_class)

loss.backward()

optimizer.step()

5. Prediction for test data

Once the model has been trained, we can test it on our test dataset to see the performance of the model. We take the last output scores of classes to get the final prediction regarding the sentiment from the tweet. The maximum value of the scores is the predicted class. We append those to the list for finally evaluating our model performance.

# Convert the sentiment_class from set to list

sentiment_class = list(sentiment_class)



y_pred = []

y_actual = []

with torch.no_grad():

for i in range(len(test_tweets)):

sentence = test_tweets[i]

sent_class = tweet_sent_class[9000+i]

inputs = prepare_sequence(sentence)

hidden = rnn.init_hidden()

for i in range(len(inputs)):

class_scores, hidden = rnn(inputs[i], hidden)

# for word i. The predicted tag is the maximum scoring tag. y_pred.append(sentiment_class[((class_scores.max(dim=1)[1].numpy()))[0]])

y_actual.append(str(sent_class))

6. Results

Using this simple model with 30 epochs and only 9000 training document, the accuracy obtained was ~63.62% and the confusion matrix is as shown in the following.

print(sentiment_class)

print(confusion_matrix(y_actual, y_pred, labels=sentiment_class))

print(accuracy_score(y_actual, y_pred))

Results obtained:

['negative', 'positive', 'neutral']

[[2980 291 935]

[ 203 255 143]

[ 373 107 353]]

0.6361702127659574

Certainly, this result can be improved with more documents, using different hidden and embedding size choices and number of epochs. The stopwords that were removed can also be kept instead as it may indicate better intuition of the sequence and not to forget as they are tweets, it is filled with special characters and tagging so removing that would also improve the quality of your input. We used variable sized inputs for this experiment(length of sentences), padding the input vector can be used to improve the training as well. RNNs can only store little context and hence using LSTMs and GRUs for these tasks with longer contexts are more popular.

The complete code and output file of predictions can be found here. At the end I have mentioned some references which helped me on the path of learning this.

Hope you find this useful in your journey of learning and exploring !! :)

References:

Deep learning for Sentiment Analysis: https://arxiv.org/pdf/1801.07883.pdf

2. LSTM: http://colah.github.io/posts/2015-08-Understanding-LSTMs/

3. Backpropagation: http://colah.github.io/posts/2015-08-Backprop/

4. Pytorch Character level classification with RNN: https://pytorch.org/tutorials/intermediate/char_rnn_classification_tutorial.html

5. RNNs from scratch in python (Intuitive visualization): https://iamtrask.github.io/2015/11/15/anyone-can-code-lstm/

Author: Dipika Baad