Reading between the layers (LSTM Network) | by Samarth Agrawal | Towards Data Science

One of the most crucial part of building a Deep Neural Network is— to have a clear view on your data as it flows through layers undergoing change in dimensions, alteration in shape, flattening and then re-shaping...
We will refer to the LSTM Architecture that we have seen earlier in our Sentiment Analysis Tutorial. Link to the article here.
towardsdatascience.com
The layers are as follows:
0. Tokenize : This is not a layer for LSTM network but a mandatory step of converting our words into tokens (integers)
Embedding Layer: that converts our word tokens (integers) into embedding of specific sizeLSTM Layer: defined by hidden state dims and number of layersFully Connected Layer: that maps output of LSTM layer to a desired output sizeSigmoid Activation Layer: that turns all output values in a value between 0 and 1Output: Sigmoid output from the last timestep is considered as the final output of this network
Embedding Layer: that converts our word tokens (integers) into embedding of specific size
LSTM Layer: defined by hidden state dims and number of layers
Fully Connected Layer: that maps output of LSTM layer to a desired output size
Sigmoid Activation Layer: that turns all output values in a value between 0 and 1
Output: Sigmoid output from the last timestep is considered as the final output of this network
Before you define the Model Class it will give a good insight to have a closer look at each of the layers. This will help you get more clarity on how to prepare your inputs for Embedding, LSTM, Linear layers of your model architecture
We are using IMDB movies review dataset, data processing and preparation steps are already done. Click here, if you need to revisit those steps. We are starting off with dataloaders (where we have defined batch_size=50 and sequence length=200). As per my article on building Sentiment Analysis model using LSTM, we are looking at Step 14 with a microscope —
Let’s start by looking at inupts and targets from dataloaders :
dataiter = iter(train_loader)x, y = dataiter.next()x = x.type(torch.LongTensor)print ('X is', x)print ('Shape of X and y are :', x.shape, y.shape)
From the shape of X we can see that X is a tensor of 50 rows (=batch size) & 200 columns (=sequence length). This assures that our process of tokenization is working fine. This X will go as input into Embedding layer
Embedding layer :
The module that allows you to use Embedding is torch.nn.Embedding. It takes two parameters : the vocabulary size and the dimensionality of the embedding
from torch import nnvocab_size = len(words)embedding_dim = 30embeds = nn.Embedding(vocab_size, embedding_dim)print ('Embedding layer is ', embeds)print ('Embedding layer weights ', embeds.weight.shape)
embeds_out = embeds(x)print ('Embedding layer output shape', embeds_out.shape)print ('Embedding layer output ', embeds_out)
From the output of embedding layer we can see it has created a 3 dimensional tensor as a result of embedding weights. Now it has 50 rows, 200 columns and 30 embedding dimension i.e. for each tokenized word in our review we have added embedding dimension. This data will now go to LSTM Layer
LSTM Layer :
While defining the LSTM layer we have kept Batch First = True and number of hidden units = 512.
# initializing the hidden state to 0hidden=Nonelstm = nn.LSTM(input_size=embedding_dim, hidden_size=512, num_layers=1, batch_first=True)lstm_out, h = lstm(embeds_out, hidden)print ('LSTM layer output shape', lstm_out.shape)print ('LSTM layer output ', lstm_out)
By looking at the output of LSTM layer we see that our tensor is now has 50 rows, 200 columns and 512 LSTM nodes. Next this data is fetched into Fully Connected layer
Fully Connected Layer :
For fully connected layer, number of input features = number of hidden units in LSTM. Output Size = 1 because we only binary outcome (1/0; Positive/Negative)
Note that before putting the lstm output into fc layer it has to be flattened out.
fc = nn.Linear(in_features=512, out_features=1)fc_out = fc(lstm_out.contiguous().view(-1, 512))print ('FC layer output shape', fc_out.shape)print ('FC layer output ', fc_out)
Sigmoid Activation Layer :
This is needed just to turn all output value from fully connected layer into a value between 0 and 1
sigm = nn.Sigmoid()sigm_out = sigm(fc_out)print ('Sigmoid layer output shape', sigm_out.shape)print ('Sigmoid layer output ', sigm_out)
Final Output :
This includes two steps: First, to reshape the output such that rows = batch size
batch_size = x.shape[0]out = sigm_out.view(batch_size, -1)print ('Output layer output shape', out.shape)print ('Output layer output ', out)
Second, as we see in the Network Architecture — we only want the output after the last sequence (after the last timestep)
print ('Final sentiment prediction, ', out[:,-1])
These outputs are from an untrained network and hence the values might not indicate anything yet. This was just for the sake of illustration and we will use this knowledge to define the model correctly.
I hope you have had as much fun reading it as I had writing in this piece
Try to replicate it the process for any other deep learning model you are trying to implement
Feel free to write your thoughts / suggestions / feedback