Every thing you need to know about LSTMs in Keras
Tags: Machine Learning, Data Science, NLP, Lstm

Every thing you need to know about LSTMs in Keras Ashwinkaurav Follow Sep 18 · 3 min read

Introduction

LSTMs were in existence for a very long but came into popularity due to their work in NLP i.e. Seq2Seq models which lead to neural machine translation (ex: Language Translators and Chatbots) . The struggle I had in understanding LSTMs (thanks to Keras for their poor explanation of some of the terms in documentation) made me write this blog for those who are going through a similar phase @_@. I promise you IT AINT GONNA BE TOUGH!

This blog is moreover related to how LSTMs work.

Structure

Credit: Colah’s blog on LSTM.

The first thing to note is that unlike other neural networks LSTM takes input in timesteps and every timestep there is an output i.e. Ht-1 , Ht , Ht+1 which is known as its hidden state, we generally refer to the last hidden state as Output.

Hidden State

And there is one more thing to notice that is the cell state , at every time step there is a cell state. Look at the figure below to know what is a cell state.

Cell State Ct.

To know the mathematics and more information on why these cell states and hidden states are used and what are they I would suggest you to go through the links I have cited at the end.

Implementation in Keras

The input should always be in the shape (batch_size,time_steps,features)

input = Input(shape=(time_steps,features))

lstm = LSTM(units=10)

output = lstm(input)

There is a general misconception about the “units” argument (even I had that at first) , most of the people at first assumes that the units means the number of cells in our LSTM , but that’s not the case.

The argument ‘units’ actually decides the output shape of the LSTM , which means that all the hidden states and cell states will be of shape (units , ).

Number of lstm cells is automatically decided by the input shape we give in the input layer.

Flow of data :

Shape of data at different positions

Ht-1 is of shape (units,) which we input as an argument ‘units’.

Xt is of shape (num_features , ) , Ht-1 and Xt are concatenated , so the new shape becomes (units+num_features , ).

All the weights W1,W2,W3,W4 will be of shape (units , units+num_features) .

Shape of W5=(units , units )

And the biases B1,B2,B3,B4,B5 will be of shape (units , ).

After matrix multiplication : W*[num_features,units]

(units , units+num_features) * (units+num_features , )

Shape = (units , )

We will only get the final output i.e. hidden state at last step

2. return_sequences

When return_sequences is set to True in argument , the lstm will output a list of the hidden states at each time step rather than the hidden state at last time step. By default it is set to False.

input = Input(shape=(time_steps,features))

lstm = LSTM(units=10,return_sequences=True)

output = lstm(input)

Output shape will be ( time_steps , units )

3. return_state

When return_state is set to True in argument , the lstm will output both hidden_state , cell_state of the last timestep. Unfortunately there is no way known to me by which we can get cell_state at each timestep.

And I bet you know what would be the shape of both ;-).

input = Input(shape=(time_steps,features))

lstm = LSTM(units=10,return_state=True)

hidden_state,cell_state = lstm(input)

4. return_state,return_sequences