Sentiment Analysis using Word embeddings with Tensorflow
By: Aditya Kousik Cotra
Tags: Deep Learning, Machine Learning, TensorFlow, Sentiment Analysis, Data Science

Photo by Markus Winkler on Unsplash

Sentiment Analysis using Word embeddings with Tensorflow

Classifying sentiment in IMDB reviews

In this blog post, we will train word embeddings on the IMBD reviews dataset using Tensorflow.

First, we will go through the known strategies of representing texts as numbers and what actually word embeddings are and how they are useful in classifying and understanding texts far more efficiently than other word representations. Then we will train our neural networks and classify reviews.

Representing text as numbers

Machine learning models take numerical vectors as input. So the first thing is to come up with a strategy to vectorize texts before feeding it to the model. Let’s look at the three strategies for doing so.

One-hot encoding:

Let’s say we have a vocabulary of 10,000 words. So for example, if man is word number 4342 in this dictionary, then you represent him with a vector with one in position 4342. To represent each word, we will create a zero vector with length equal to the vocabulary, then place a one in the index that corresponds to the word. For a sentence, we concatenate encodings of all the words.

One-hot encoding representation by The Tensorflow Authors

This approach does not capture any relationship between words. It merely shows that the word is present in the given sentence.

Encode each word with a unique number:

Continuing the example above, we could assign 1 to “cat”, 2 to “mat”, and so on. We could then encode the sentence “The cat sat on the mat” as a dense vector like [5, 1, 4, 3, 5, 2]. This appoach is efficient. Instead of a sparse vector, we now have a dense one (where all elements are full).

Even this approach does not capture any relationship between words.

Word Embeddings

Word embeddings are dense representation of words where similar words have similar meanings. It is similar to a word having different features.

The table below shows a sample word embedding representation

Word embedding representations

Look at the features Gender, Royal, Age, Food and Company and how they represent each word. A 5-dimension word embedding for the word ‘king’ from the above representation is:

vec[‘king’] = [0.97,0.99,0.7,0.04,0.001]

This representation captures the semantic relations effectively. We can perform computations like

vec[‘queen’]+vec[‘man’]-vec[‘woman’] = vec[‘king’]

It is common to see word embeddings that are 8-dimensional (for small datasets), up to 1024-dimensions when working with large datasets. A higher dimensional embedding can capture fine-grained relationships between words, but takes more data to learn.

Word embeddings on IMDB Dataset

The IMDB dataset consists of reviews with positive (1) or negative (0) labels.

We will go through a step-by-step process of learning word embeddings using tensorflow. First, we import the necessary libraries and load the imdb reviews dataset using the tensorflow_datasets module. Then we will preprocess the dataset using tokenizing and padding on the data. Then we will create a model using tensorflow.keras.Sequential with an embedding layer and one hidden layer.

The notebook for this article can be found here.

Import Libraries

import numpy as np

import tensorflow as tf

import tensorflow_datasets as tfds

from tensorflow.keras.preprocessing.text import Tokenizer

from tensorflow.keras.preprocessing.sequence import pad_sequences

Load dataset

Now, we load the IMDB reviews dataset from the tensorflow_datasets module.

imdb = tfds.load('imdb_reviews',as_supervised=True)

train, test = imdb['train'], imdb['test']

Preprocessing

We need to maintain the sentences and labels in a list.Then we input the sentence lists to the tokenizer to form sequences.

training_sentences=[]

training_labels=[]

testing_sentences=[]

testing_labels=[] for sentence,label in train_data:

training_sentences.append(str(sentence.numpy()))

training_labels.append(label.numpy())

for sentence,label in test_data:

testing_sentences.append(str(sentence.numpy()))

testing_labels.append(label.numpy()) training_labels_final=np.array(training_labels)

testing_labels_final=np.array(testing_labels)

The values of sentence and label in the for loop are tensors, so we need to convert it using s.numpy() . At the end, we convert labels to numpy array.

Now we will first define the the parameters.

vocab_size=10000

embedding_dim = 16

max_length = 120

trunc_type='post'

oov_tok = "<OOV>"

The oov_token is defined to label the out-of-vocabulary words.

Now we tokenize the data.

tokenizer = Tokenizer(num_words = vocab_size, oov_token=oov_tok)

tokenizer.fit_on_texts(training_sentences)

word_index = tokenizer.word_index

The word_index contains the key-value pairs of words and their index in the vocabulary. Now we convert the sentences in training_sentences into sequence of indices.

training_sequences=tokenizer.texts_to_sequences(training_sentences)

testing_sequences=tokenizer.texts_to_sequences(testing_sentences)

We pad the sequences using pad_sequences so that all the sentences are of equal lengths of max_length when feeding into the model. We can choose the truncating parameter to pre or post . The default value is pre but we will use post here to truncate the end of the sentences.

training_padded=pad_sequences(training)_sequences,maxlen=max_length,truncating=trunc_type) testing_padded=pad_sequences(testing_sequences,maxlen=max_length,truncating=trunc_type)

Now our data is all set to be fed into the model.

Build and train Neural Network

We build define our neural network model using tf.keras.Sequential and by adding layers to it as follows.

model = tf.keras.Sequential([

tf.keras.layers.Embedding(vocab_size, embedding_dim, input_length=max_length),

tf.keras.layers.Flatten(),

tf.keras.layers.Dense(6, activation='relu'),

tf.keras.layers.Dense(1, activation='sigmoid')

]) model.compile(loss='binary_crossentropy',optimizer='adam',metrics=['accuracy']) model.summary()

Model Summary

The Embedding layer input output dimensions is set to (vocab_size, embedding_dim) and the input_length is set to max_length since the input sequences are padded to max_length. We will use binary_crossentropy as our loss metric and collect accuracy metrics on train and validation data.

Now, we train the model with 10 epochs. We use the testing sequences and testing labels for validation of the model.

model.fit(training_padded, training_labels_final, epochs=10, validation_data=(testing_padded, testing_labels_final))

With this approach our model reaches a validation accuracy of around 83% (note the model is overfitting, training accuracy is significantly higher).

Now we get our word embeddings.

e = model.layers[0]

weights = e.get_weights()[0]

print(weights.shape) # shape: (vocab_size, embedding_dim)

Predict new reviews

Let’s create a list of new reviews, preprocess and see how our model predicts.

new_sents = [

'This movie was awesome. Really loved it.',

'I almost fell asleep watching this.',

'This movie is so stupid.',

'I wouldn\'t watch this movie unless there\'s a gun pointed at me.',

'This movie is so hilarious. Much love to the director!'

] new_seq = tokenizer.texts_to_sequences(new_sents)

padded=pad_sequences(new_seq, maxlen=max_length,truncating=trunc_type)

output=model.predict(padded)

for i in range(0,len(new_sents)):

print('Review:'+new_sents[i]+' '+'sentiment:'+str(output[i])+'

')

Looks like our model worked well!

Conclusion

Text classification using word embeddings can be done in other ways using other libraries like Word2Vec or FastText. We will look at it in the coming articles.