SEQ2SEQ LEARNING. PART F: Encoder-Decoder with Bahdanau &…
By: Murat Karakaya
Tags: Bahdanau, Encoder Decoder, Attention, Keras, Murat Karakaya Akademi

In an Encoder-Decoder with Attention Layer set-up,

Encoder provides:

the initial states by sending its last hidden state + last cell state

by sending its the context vector by sending its all hidden states

The decoder needs 2 inputs to generate/predict an output:

an input tensor

a state tensor

The decoder:

initializes its state by consuming the ***initial state***s

uses decoder’s last hidden state as the initial input

as the calculates attention vector using initial input + encoder’s all hidden states

using + applies the attention to encoder’s all hidden states finds the context vector

to finds the concatenate context vector + START to generate the decoder input

+ to generate the then runs in a loop:

consume the input and states

outputs its last hidden state, last hidden state (yes again!), last cell state ,

, use last hidden state + last cell state as the next state

as the use last hidden state as the next initial input

as the next calculates attention vector using initial input + encoder’s all hidden states

using + applies the attention to encoder’s all hidden states finds the context vector

finds the concatenate context vector+ initial input to generate the decoder input

verbose= 0

#See all debug messages



batch_size=1

if verbose:

print('***** Model Hyper Parameters *******')

print('latentSpaceDimension: ', latentSpaceDimension)

print('batch_size: ', batch_size)

print('sequence length: ', n_timesteps_in)

print('n_features: ', n_features)



print('

***** TENSOR DIMENSIONS *******')



# The first part is encoder

encoder_inputs = Input(shape=(n_timesteps_in, n_features), name='encoder_inputs')

encoder_lstm = LSTM(latentSpaceDimension,return_sequences=True, return_state=True, name='encoder_lstm')

encoder_outputs, encoder_state_h, encoder_state_c = encoder_lstm(encoder_inputs)



if verbose:

print ('Encoder output shape: (batch size, sequence length, latentSpaceDimension) {}'.format(encoder_outputs.shape))

print ('Encoder Hidden state shape: (batch size, latentSpaceDimension) {}'.format(encoder_state_h.shape))

print ('Encoder Cell state shape: (batch size, latentSpaceDimension) {}'.format(encoder_state_c.shape))

# initial context vector is the states of the encoder

encoder_states = [encoder_state_h, encoder_state_c]

if verbose:

print(encoder_states)

# Set up the attention layer

attention= BahdanauAttention(latentSpaceDimension, verbose=verbose)





# Set up the decoder layers

decoder_inputs = Input(shape=(1, (n_features+latentSpaceDimension)),name='decoder_inputs')

decoder_lstm = LSTM(latentSpaceDimension, return_state=True, name='decoder_lstm')

decoder_dense = Dense(n_features, activation='softmax', name='decoder_dense')



all_outputs = []



# 1 initial decoder's input data

# Prepare initial decoder input data that just contains the start character

# Note that we made it a constant one-hot-encoded in the model

# that is, [1 0 0 0 0 0 0 0 0 0] is the first input for each loop

# one-hot encoded zero(0) is the start symbol

inputs = np.zeros((batch_size, 1, n_features))

inputs[:, 0, 0] = 1





# 2 initial decoder's state

# encoder's last hidden state + last cell state

decoder_outputs = encoder_state_h

states = encoder_states

if verbose:

print('initial decoder inputs: ', inputs.shape)



# decoder will only process one time step at a time.

for _ in range(n_timesteps_in):



# 3 pay attention

# create the context vector by applying attention to

# decoder_outputs (last hidden state) + encoder_outputs (all hidden states)

context_vector, attention_weights=attention(decoder_outputs, encoder_outputs)

if verbose:

print("Attention context_vector: (batch size, units) {}".format(context_vector.shape))

print("Attention weights : (batch_size, sequence_length, 1) {}".format(attention_weights.shape))

print('decoder_outputs: (batch_size, latentSpaceDimension) ', decoder_outputs.shape )



context_vector = tf.expand_dims(context_vector, 1)

if verbose:

print('Reshaped context_vector: ', context_vector.shape )



# 4. concatenate the input + context vectore to find the next decoder's input

inputs = tf.concat([context_vector, inputs], axis=-1)



if verbose:

print('After concat inputs: (batch_size, 1, n_features + hidden_size): ',inputs.shape )



# 5. passing the concatenated vector to the LSTM

# Run the decoder on one timestep with attended input and previous states

decoder_outputs, state_h, state_c = decoder_lstm(inputs,

initial_state=states)

#decoder_outputs = tf.reshape(decoder_outputs, (-1, decoder_outputs.shape[2]))



outputs = decoder_dense(decoder_outputs)

# 6. Use the last hidden state for prediction the output

# save the current prediction

# we will concatenate all predictions later

outputs = tf.expand_dims(outputs, 1)

all_outputs.append(outputs)

# 7. Reinject the output (prediction) as inputs for the next loop iteration

# as well as update the states

inputs = outputs

states = [state_h, state_c]





# 8. After running Decoder for max time steps

# we had created a predition list for the output sequence

# convert the list to output array by Concatenating all predictions

# such as [batch_size, timesteps, features]

decoder_outputs = Lambda(lambda x: K.concatenate(x, axis=1))(all_outputs)



# 9. Define and compile model

model_encoder_decoder_Bahdanau_Attention = Model(encoder_inputs, decoder_outputs, name='model_encoder_decoder')

model_encoder_decoder_Bahdanau_Attention.compile(optimizer='rmsprop', loss='categorical_crossentropy', metrics=['accuracy'])

TRAIN THE MODEL WITH ATTENTION

train_test(model_encoder_decoder_Bahdanau_Attention, X_train, y_train , X_test,

y_test, batch_size=batch_size,epochs=40, patience=3, verbose=1) training for 40 epochs begins with EarlyStopping(monitor= val_accuracy, patience= 3 )....

Epoch 1/40

1800/1800 [==============================] - 19s 7ms/step - loss: 2.0012 - accuracy: 0.2725 - val_loss: 1.4980 - val_accuracy: 0.4025

***

Epoch 7/40

1800/1800 [==============================] - 11s 6ms/step - loss: 5.6494e-04 - accuracy: 0.9999 - val_loss: 1.1552e-06 - val_accuracy: 1.0000

Epoch 8/40

1800/1800 [==============================] - 11s 6ms/step - loss: 7.0385e-05 - accuracy: 1.0000 - val_loss: 1.7178e-07 - val_accuracy: 1.0000

Epoch 00008: early stopping

40 epoch training finished...



PREDICTION ACCURACY (%):

Train: 100.000, Test: 100.000

10 examples from test data...

Input Expected Predicted T/F

[2, 3, 1, 1] [1, 1, 3, 2] [1, 1, 3, 2] True

[7, 9, 1, 6] [6, 1, 9, 7] [6, 1, 9, 7] True

[2, 9, 3, 8] [8, 3, 9, 2] [8, 3, 9, 2] True

[7, 7, 9, 9] [9, 9, 7, 7] [9, 9, 7, 7] True

[7, 1, 1, 7] [7, 1, 1, 7] [7, 1, 1, 7] True

[8, 4, 6, 9] [9, 6, 4, 8] [9, 6, 4, 8] True

[2, 5, 9, 9] [9, 9, 5, 2] [9, 9, 5, 2] True

[2, 2, 4, 5] [5, 4, 2, 2] [5, 4, 2, 2] True

[8, 7, 5, 7] [7, 5, 7, 8] [7, 5, 7, 8] True

[4, 9, 9, 1] [1, 9, 9, 4] [1, 9, 9, 4] True

Accuracy: 1.0

time: 1min 38s

Observations

When the sequence size ( n_timesteps_in ) is 4

Encoder-Decoder model terminates at Epoch 31 with 99% accuracy .

. Encoder-Decoder Model with Attention terminates at Epoch 9 with 100%

However, when the sequence size ( n_timesteps_in ) is 16

Encoder-Decoder model runs all the 40 epochs and finishes with only 36% accuracy!.

and finishes with only accuracy!. Encoder-Decoder Model with Attention terminates at Epoch 16 with 99%

We can conclude that the Encoder-Decoder model with Attention is much more scalable in terms of sequence length.

PREDICT WITH THE TRAINED MODEL