Sentences classification using RNN
By: Vishnu Gopireddy
Tags: Sentiment Analysis, Machine Learning, Recurrent Neural Network, Lstm

RNN including LSTM’s, GRU’s are most popular architectures for processing sequential data. A sentence is nothing but a sequence of words, RNN’s can be used for sentence classification and a simple RNN can give an with just one LSTM layer we can get acuuracy over 90% !!!

Here we use LSTM’s to perform our task. LSTM’s are very detailedly explained here. Many to one RNN architecture suits for our task, since we have a sentence(sequence of words) are to be processed and predict as positive or negative.

Various architectures of RNN

For illustration, we use IMDB reviews dataset in keras. Each review is encoded as a sequence of word indexes. Keras has great documentation please refer it for more detailed information.

#load dataset, include only top 5K words

(X_train, y_train),(X_test, y_test)=imdb.load_data(num_words=5000)

#A sample review in both text and encoded form

Encoded : [1, 13, 586, 851, 14, 31, 60, 23, 2863, 2364, 314]

Text : <START> i wouldn't rent this one even on dollar rental night

Now the problem is that the length of the reviews are not even, so we can’t perform batch SGD. So the next step is to make the length of all reviews equal. One simple way to do is pad zero’s in the beginning and prune last words for long reviews(above 500 words).

# Makes each review of size 500 words

X_train = sequence.pad_sequences(X_train,maxlen=500,padding='pre')

X_test = sequence.pad_sequences(X_test,maxlen=500,padding='pre')

Next step is to build LSTM(RNN), model. But a neural network doesn’t understand words directly, so we add an embedding layer before LSTM layers which converts words to embeddings.

We can think of the embedding layer just as word2vec but the difference is embedding layer performs supervised learning to get word embeddings whereas word2vec gives static vectors. Embedding layer vectors vary upon the corpus, whereas word2vec representation is static and pre-learned.

# Build model

size_of_vocab = 5000

word_size = 500 model = Sequential()

model.add(Embedding(size_of_vocab,32,input_length=word_size))

model.add(LSTM(100))

model.add(Dense(1,activation='sigmoid'))

For a given word, embedding layer gives a 32 Dim vector and is given to 100 LSTM units. For the next word along with 32 dim input, output of the previous word is taken as a hidden state parallelly for 100 LSTM units. This keeps on. At 500th word we end up having 100 LSTM outputs connected to a sigmoid unit for prediction. Summary of the model looks like bleow.

model.summary() _________________________________________________________________

Layer (type) Output Shape Param #

=================================================================

embedding_1 (Embedding) (None, 500, 32) 160000

_________________________________________________________________

lstm_1 (LSTM) (None, 100) 53200

_________________________________________________________________

dense_1 (Dense) (None, 1) 101

=================================================================

Total params: 213,301

Trainable params: 213,301

Non-trainable params: 0

mmodel = multi_gpu_model(model,gpus=2)

model.compile('RMSprop',loss='binary_crossentropy',metrics=['acc'])

tboard = TensorBoard(log_dir='/mnt/HDD/vishnu/lstm/')

early = EarlyStopping(monitor='val_loss',patience=5)

#tboard = tensorboard(log)

Use RMSProp optimizer to reduce binary cross entropy and monitor accuracy for each epoch.

I have got two GPU’s and can use multi model gpu for training on two GPU’s.

Early stoppage to stop training at right time and epoch.

Tensorboard for monitoring loss and accuracy.

model.fit(x=X_train,y=y_train,batch_size=64,epochs=3,validation_split=0.2,callbacks=[tboard,early]) Train on 20000 samples, validate on 5000 samples

Epoch 1/3

20000/20000 [==============================] - 283s 14ms/step - loss: 0.5890 - acc: 0.6945 - val_loss: 0.4799 - val_acc: 0.7788

Epoch 2/3

20000/20000 [==============================] - 282s 14ms/step - loss: 0.4396 - acc: 0.8046 - val_loss: 0.4136 - val_acc: 0.8206

Epoch 3/3

20000/20000 [==============================] - 284s 14ms/step - loss: 0.3952 - acc: 0.8307 - val_loss: 0.3912 - val_acc: 0.8318

Fit the model with batch size 64 and 3 epochs. With just 3 epochs we can get accuracy over 83%, it can be improved after tuning hyper parameters.

Full code is available on my github.