A Beginner’s Guide on Sentiment Analysis with RNN | by Susan Li | Towards Data Science

Sentiment analysis probably is one the most common applications in Natural Language processing. I don’t have to emphasize how important customer service tool sentiment analysis has become. So here we are, we will train a classifier movie reviews in IMDB data set, using Recurrent Neural Networks. If you want to dive deeper on deep learning for sentiment analysis, this is a good paper.
We will use Recurrent Neural Networks, and in particular LSTMs, to perform sentiment analysis in Keras. Conveniently, Keras has a built-in IMDb movie reviews data set that we can use.
from keras.datasets import imdb
Set the vocabulary size and load in training and test data.
vocabulary_size = 5000(X_train, y_train), (X_test, y_test) = imdb.load_data(num_words = vocabulary_size)print('Loaded dataset with {} training samples, {} test samples'.format(len(X_train), len(X_test)))
Loaded dataset with 25000 training samples, 25000 test samples
Inspect a sample review and its label.
print('---review---')print(X_train[6])print('---label---')print(y_train[6])
Note that the review is stored as a sequence of integers. These are word IDs that have been pre-assigned to individual words, and the label is an integer (0 for negative, 1 for positive).
We can use the dictionary returned by imdb.get_word_index() to map the review back to the original words.
word2id = imdb.get_word_index()id2word = {i: word for word, i in word2id.items()}print('---review with words---')print([id2word.get(i, ' ') for i in X_train[6]])print('---label---')print(y_train[6])
Maximum review length and minimum review length.
print('Maximum review length: {}'.format(len(max((X_train + X_test), key=len))))
Maximum review length: 2697
print('Minimum review length: {}'.format(len(min((X_test + X_test), key=len))))
Minimum review length: 14
In order to feed this data into our RNN, all input documents must have the same length. We will limit the maximum review length to max_words by truncating longer reviews and padding shorter reviews with a null value (0). We can accomplish this using the pad_sequences() function in Keras. For now, set max_words to 500.
from keras.preprocessing import sequencemax_words = 500X_train = sequence.pad_sequences(X_train, maxlen=max_words)X_test = sequence.pad_sequences(X_test, maxlen=max_words)
We start building our model architecture in the code cell below. We have imported some layers from Keras that you might need but feel free to use any other layers / transformations you like.
Remember that our input is a sequence of words (technically, integer word IDs) of maximum length = max_words, and our output is a binary sentiment label (0 or 1).
from keras import Sequentialfrom keras.layers import Embedding, LSTM, Dense, Dropoutembedding_size=32model=Sequential()model.add(Embedding(vocabulary_size, embedding_size, input_length=max_words))model.add(LSTM(100))model.add(Dense(1, activation='sigmoid'))print(model.summary())
To summarize, our model is a simple RNN model with 1 embedding, 1 LSTM and 1 dense layers. 213,301 parameters in total need to be trained.
We first need to compile our model by specifying the loss function and optimizer we want to use while training, as well as any evaluation metrics we’d like to measure. Specify the appropriate parameters, including at least one metric ‘accuracy’.
model.compile(loss='binary_crossentropy',              optimizer='adam',              metrics=['accuracy'])
Once compiled, we can kick off the training process. There are two important training parameters that we have to specify — batch size and number of training epochs, which together with our model architecture determine the total training time.
Training may take a while, so grab a cup of coffee, or better, go for a run!
batch_size = 64num_epochs = 3X_valid, y_valid = X_train[:batch_size], y_train[:batch_size]X_train2, y_train2 = X_train[batch_size:], y_train[batch_size:]model.fit(X_train2, y_train2, validation_data=(X_valid, y_valid), batch_size=batch_size, epochs=num_epochs)
Once we have trained our model, it’s time to see how well it performs on unseen test data.
scores[1] will correspond to accuracy if we pass metrics=[‘accuracy’]
scores = model.evaluate(X_test, y_test, verbose=0)print('Test accuracy:', scores[1])
Test accuracy: 0.86964
There are several ways in which we can build our model. We can continue trying and improving the accuracy of our model by experimenting with different architectures, layers and parameters. How good can we get without taking prohibitively long to train? How do we prevent overfitting?
The source code can be found at Github. Look forward to hearing feedback or questions.
Reference: Udacity — NLP