Sentiment Analysis with TensorFlow Hub
By: Tirendaz Academy
Tags: Machine Learning, Deep Learning, Tensorflow Hub, Sentiment Analysis, Artificial Intelligence

Hands-on Tutorial

Sentiment Analysis with TensorFlow Hub

In this article, I explained sentiment analysis with TensorFlow Hub using the IMDB dataset. Tirendaz Academy Follow Jun 17 · 10 min read

Photo by Everton Vila on Unsplash

While performing data analysis, the model is trained with the training data and then inference on new data according to this trained model.

Model training is a difficult process. A large amount of labeling and computing power is required to train a model from scratch. For example, thousands of hours of GPU were run to train the NASNet model developed in 2017.

Fortunately, there are pre-trained models we can use for analysis such as text or image classification. These trained models are also available in libraries such as TensorFlow. To use these models, all you have to do is call these models and apply them to your datasets.

You may not get good results by applying these models directly to your datasets. Because these models are trained on certain data. For example, you want to classify pictures of dogs and cats. You want to use the ResNet model. ResNet is trained to classify thousands of images. You can only classify dog ​​and cat images by customizing the last layers of this model.

For more information on basic image classification, you can read the following blog post.

What is TensorFlow Hub?

TensorFlow Hub is a repository of pre-trained machine learning models. You can use models like BERT in TensorFlow Hub for your work with a few lines of code. Isn’t it great? With TensorFlow Hub, you can perform analytics such as image classification, text embeddings, audio, and video recognition.

In summary, using pre-trained models in TensorFlow Hub, you can quickly train your model and deploy the model you train for less data, less processing power, and less time for the specific area you are working on.

In this article, I will show you how to use TensorFlow Hub for step-by-step text classification using the IMDB dataset.

Sentiment Analysis

With the development of social media and technology, most people shared their ideas and thoughts. This shared data is very important for fields such as social sciences and marketing.

Analyzing whether a text is positive or negative is called sentiment analysis. Sentiment analysis is a subfield of natural language processing and is one of the popular fields today.

I will use the IMDB dataset to demonstrate Sentiment analysis with TensorFlow Hub. Let’s take a closer look at the IMDB dataset

IMDB Dataset

The IMDB dataset consists of 50,000 movie reviews, of which 25,000 are tutorials and 25,000 tests. Half of these comments were labeled as positive and the other half as negative. It is a balanced dataset as the positive and negative labels in this dataset are equal.

You will see the use of this dataset very often in text analysis courses. TensorFlow has a collection of ready-to-use datasets that we can use. With the TensorFlow Datasets (TFDS) library, you can load these datasets and make them available for analysis.

You can load this library with one of the following codes:

pip install tensorflow-datasets #for stable version

pip install tfds-nightly #Contains the latest version of datasets

After installing this library, let’s import it and tensorflow.

import tensorflow as tf

import tensorflow_datasets as tfds

The easiest way to load the IMDB dataset is to use the tfds.load() method.

Download the IMDB Dataset

Now let’s load the IMDB dataset using the tfds.load() method. While loading the dataset, let’s also split it into training, validation, and testing.

The model is fitted with the training data, the hyper-parameters are tune with the validation data, and the model is evaluated with the test data.

I want to split the training dataset into 60% training and 40% validation.

train_data, validation_data, test_data = tfds.load(

name="imdb_reviews", #1

split=('train[:60%]', 'train[60%:]', 'test'), #2

as_supervised=True #3

)

Let’s go over these codes.

(1) I set the name of the dataset in the name argument.

(2) I specified how to split the dataset to the split argument.

(3) By setting the as_supervised argument to True, I have set the structure of the dataset as input and label.

Thus, we have loaded the dataset. Let’s get to explore the dataset a little bit.

Explore the Dataset

Understanding the dataset is one of the important stages of data analysis. Each example in the IMDB dataset consists of a movie review and a label for that movie review. Labels consist of 0 or 1’s. 0 indicates the comment is negative and 1 indicates the comment is positive.

Let’s print the first 10 examples on the screen. First of all, I am going to separate the comments and tags in the examples.

train_examples_batch, train_labels_batch =

next(iter(train_data.batch(10)))

I selected the first 10 samples with the batch method I used here, iterated the components in the data set with the iter() method, and assign the components to train_examples_batch and train_labels_batch variables with the next method. Let’s see the first 10 comments.

train_examples_batch

First 10 Reviews in the Training Dataset

Let’s print the first 10 labels.

train_labels_batch

First 10 labels in the Training Dataset

Build the Model

We are now familiar with datasets. After that we can build a neural network model. Deep neural network (DNN) models have become very popular especially in recent years. You can obtain high accuracy using models based on DNN.

The biggest disadvantage of neural networks is tuning the model hyper-parameters. You won’t get good models if you don’t fine-tune the hyper-parameters. When constructing a neural network, it is necessary to specify hyper-parameters such as how the text will be represented, how many layers in the model, and how many neurons will be used in each layer.

Our input data consists of sentences. One way to represent these texts is to translate sentences into embeddings vectors. Thus, the location of a word in vector space is learned from the text. The position in the vector space from which the word is learned is called embedding. Remember that words closer in vector space have similar meanings.

The first layer in text classification must pre-trained text embedding. With this layer, text preprocessing is done. Thus, the text preprocessing process, which is one of the most tiring processes, is easily done.

There are many pre-trained text embeddings on TFHub. These text embeddings are based on feed-forward Neural-Net Language Models (NNLM). You can use one of these embeddings according to your work.

I will use nnlm-en-dim50/2 in this study. This model uses tokens based on text embedding trained in 7D English google news corpus and includes 50 embedding dimensions.

If you want, you can also use the 128 embedding-dimension architecture, trained in the 200B English google news corpus named nnlm-en-dim128/2. You can get better models using larger size embeddings, but your model will take longer to train.

First, I am going to assign the pre-trained embedding layer we will use to the embedding variable.

Now let’s create a Keras layer using the TensorFlow Hub model. Let’s import tensorflow_hub with hub.

import tensorflow_hub as hub

Now let’s create a Keras layer that can be called later.

hub_layer = hub.KerasLayer(embedding, input_shape=[], dtype=tf.string, trainable=True)

In this layer, the length of the input text does not matter. The dtype argument is the data type of the layer weights. Since the data type is text, I wrote tf.string in this argument. I set True in the trainable argument because I want the layers to be trainable.

Let’s see the output of the first three samples in the training dataset according to the hub_layer layer.

hub_layer(train_examples_batch[:3])

First Three Examples of Training Dataset

Now we can build the whole model. I will use the Sequential technique to build the model.

model = tf.keras.Sequential()

I am going to add each layer with the add() method. First, let’s add the hub_layer embedding layer that I just created.

model.add(hub_layer)

With this layer, sentences are divided into tokens. Tokens that are close to each other in meaning are brought together. Next, let’s add a 16-layer Dense layer and use the relu activation function in this layer.

model.add(tf.keras.layers.Dense(16, activation='relu'))

Since our latest output has two classes, positive and negative, let’s add a 1-layer dense layer.

model.add(tf.keras.layers.Dense(1))

The whole code we wrote looks like the following.

model = tf.keras.Sequential()

model.add(hub_layer)

model.add(tf.keras.layers.Dense(16, activation=’relu’))

model.add(tf.keras.layers.Dense(1))

That’s it. We built our model.

If you want, we can use the summary() method to print the summary of the model.

model.summary()

Model Summary

You can see the number of parameters used in each layer.

Loss Function and Optimizer

Now we can compile the model. The compile() method is used to compile the model. When compiling the model, we need to specify the loss function and an optimizer. Since the problem we are considering is a classification problem, we can use binary_crossentropy as the loss function.

Let’s use the recently popular ‘adam’ as the optimizer. And let’s use the “accuracy” metric to see the performance of the model in each last epoch.

model.compile(optimizer=’adam’, loss=tf.keras.losses.BinaryCrossentropy(from_logits=True), metrics=[‘accuracy’])

For more information on these argument, you can read the blog post below.

So we compiled the model and now the model is ready for training.

Train the model

The fit() method is called to train the model. We will use the training data to train the model.

history = model.fit(

train_data.shuffle(10000).batch(512), #1

epochs=10, #2

validation_data=validation_data.batch(512), #3

verbose=1 #4

)

Training and Validation Loss and Accuracy Values

Let’s go through these codes step by step.

(1) I set the shuffle(10000) method to shuffle the training data every 10,000 samples. I wanted to be taken the samples in groups of 512 while training the model. I used the batch(512) method for this.

(2) I set 10 for the epoch number. The epoch determines how many times all data will pass through the neural network.

(3) I wrote the validation dataset in the validation_data argument. While the model is established with the training data, the hyper-parameters of the model are adjusted with the validation data.

(4) I set the verbose=1 argument to see the progress bar and information line in each epoch.

Thus, our model was trained and the accuracy and loss values ​​of the model were written to the screen in each epoch. As we expected, the training and validation accuracy of the model increases as the number of epochs increases. On the other hand, training and validation loss values ​​are decreasing.

If you notice, the accuracy of the model on the training data is more than the accuracy of the validation data. It shows that there may be an overfitting problem in this model. Models with overfitting problems have generalization problems and have difficulty in predicting new data well. You can use L1, L2, or Dropout regularization techniques to solve the model’s memorization problem.

Evaluate the Model

So far we have built the model first and then trained the model. So how does the model perform? The test dataset is used to see the performance of the model. Let’s find the accuracy value of the model on the test data by calling the evaluate method.

results = model.evaluate(test_data.batch(512), verbose=2)

Now, using the zip method, let’s print the loss and accuracy values ​​on the test data set of the model.

for name, value in zip(model.metrics_names, results):

print("%s: %.3f" %(name, value))

The accuracy of the model turned out to be about 86 percent. By using regularization techniques, you can increase the accuracy of the model to 95 percent.

Conclusion

TensorFlow Hub is a repository of pre-trained models. These models are fine-tuned and ready for deployment. You can add these models to your analysis using just a few lines of code. You can save time and processing power by using these models. With TensorFlow Hub, you can perform analytics such as image classification, text embeddings, audio, and video recognition.

In this post, I trained a machine learning model for sentiment analysis using pre-trained nnlm-en-dim128/2 embedding layer in TensorFlow Hub. The accuracy of the model I built was 99 percent on the training data set and 86 percent on the test data set.

Since the accuracy value of the model on the training dataset is greater than the accuracy value on the test dataset, the model has a memorization problem. Memorizing models cannot be generalized well. You can use regularization techniques such as dropout to overcome the memorization problem. I hope you enjoyed this article.

You can find the notebook I used in this blog post on our github page. You may be interested in the following articles.

Don’t forget to follow us on GitHub 🌱, Twitter 😎, Kaggle 📚, LinkedIn 👍

See you in the next post …