Examining The Weight And Bias of LSTM in Tensorflow 2 | by Muhammad Ryan | Towards Data Science

Ok, first, what is LSTM? LSTM is an abbreviation for Long Short Term Memory. It’s one of the Recurrent Neural Network (RNN) that most widely used. Basically, RNN takes time-series data as an input. Every single data in time series become the input of RNN sequentially from the earliest timestep. The output of RNN from the certain timestep input will be the input of the RNN together with the data of the next timestep. The real output will be outputted when the RNN finishes processing the last timestep of the data.
The structure of neuron of LSTM is like this:
In every process of the timestep, LSTM has 4 layers of the neuron. These 4 layers together forming a processing called gate called Forget gate -> Input Gate -> Output gate (-> means the order of sequence processing happens in the LSTM). And that is LSTM, I will not cover the details about LSTM because that would be a very long post and it’s not my focus this time.
Long story short, for the sake of my recent experiment, I need to retrieve the weights and biases of my LSTM. I build an LSTM using TensorFlow 2. In TensorFlow 2, we can access the structure of LSTM’s weights and biases using this code below.
import tensorflow as tfsimple_lstm_model = tf.keras.models.Sequential()simple_lstm_model.add(tf.keras.Input((18,7)))simple_lstm_model.add(tf.keras.layers.LSTM(2))print(simple_lstm_model.layers[0].trainable_weights)
In the code above, I build an LSTM that take input with shape 18 x 7. 18 is the total timesteps of the data and 7 is the total number of parameters. For every timestep, LSTM will take 7 parameters. I declare that this LSTM has 2 hidden states. The hidden state is like an output of LSTM in every timestep. This means our LSTM will be outputting 2 real numbers in the end. It also means, the amount of neurons in every layer of the neuron in LSTM is 2. When you run this script, you will get an output like this.
[<tf.Variable 'lstm/kernel:0' shape=(7, 8) dtype=float32>, <tf.Variable 'lstm/recurrent_kernel:0' shape=(2, 8) dtype=float32>, <tf.Variable 'lstm/bias:0' shape=(8,) dtype=float32>]
Ok, so here we have a 3 set of parameters and it’ consists of lstm/kernel (with shape 7x8), lstm/recurrent_kernel (with shape 2x8), and lstm/bias (with shape 8). The last part (lstm/bias) is obvious, it’s the bias. Why we have 8 of them? Remember, in the LSTM we have 4 layers of neurons and I declare this LSTM has 2 hidden states or 2 neurons in every layer. Every neuron has 1 bias, there are 2 neurons in each of 4 layers so the total is 8.
The confusing part is lstm/kernel and lstm/recurrent_kernel. Kernel here means weight. lstm/kernel means weights of our input against every neuron in LSTM. I declare the input has 18 timestep and 7 parameters, so every parameter has 1 weight against every neuron, and that’s why lstm/kernel has shape 7x8. And the last, lstm/recurrent_kernel means weights of our hidden state a.k.a the output of LSTM in the previous timestep (t-1) against every neuron in LSTM. I declare the hidden state is 2 and the total neuron in LSTM is 8, so the shape of weights of the hidden state against layers of neuron in LSTM must be 2 x 8.
And that is the explanation about weights and biases of LSTMin TensorFlow 2. You also can see the structure of the entire neural network using summary() function like below.
import tensorflow as tfsimple_lstm_model = tf.keras.models.Sequential()simple_lstm_model.add(tf.keras.Input((18,7)))simple_lstm_model.add(tf.keras.layers.LSTM(2))simple_lstm_model.add(tf.keras.layers.Dense(5))simple_lstm_model.summary()
And the output is
Model: "sequential"_________________________________________________________________Layer (type)                 Output Shape              Param #   =================================================================lstm (LSTM)                  (None, 2)                 80        _________________________________________________________________dense (Dense)                (None, 5)                 15        =================================================================Total params: 95Trainable params: 95Non-trainable params: 0
Here I stack the LSTM before with an ordinary layer of neural network. The final output of this network has 5 elements. The total of weights and biases in this stacked neural network is 95.
That’s all, see you in the next post.
Reference:
https://medium.com/ai-journal/lstm-gru-recurrent-neural-networks-81fe2bcdf1f9, accessed on 15 December 2019
https://colah.github.io/posts/2015-08-Understanding-LSTMs/, accessed on 15 December 2019