NLP | Sequence to Sequence Networks| Part 2|Seq2seq Model (EncoderDecoder Model) | by Mohammed AL-Ma'amari | Towards Data Science

In [NLP | Sequence to Sequence Networks| Part 1| Processing text data] we learnt how to process text data, In this part we will create the model which will take the data we processed and use it to train to translate English sentences to French.
We will use an architecture called (seq2seq) or ( Encoder Decoder), It is appropriate in our case where the length of the input sequence ( English sentences in our case) does not has the same length as the output data ( French sentences in our case)
Encoder decoder architecture consists of two main parts :
Encoder:
Encoder:
The encoder simply takes the input data, and train on it then it passes the last state of its recurrent layer as an initial state to the first recurrent layer of the decoder part.
Encoder input : English sentencesEncoder initial state : It depends on the initializer we use
2. Decoder :
The decoder takes the last state of encoder’s last recurrent layer and uses it as an initial state to its first recurrent layer , the input of the decoder is the sequences that we want to get ( in our case French sentences).
Decoder input : French sentencesDecoder initial state : The last state of encoder’s last recurrent layer
Some other images explaining the encoder decoder:
To learn more about encoder decoder architecture [0],[1],[2],[3],[4]
First, Import needed dependencies :
Building the encoder :
We use the (Keras Functional API) to build such complex models, if you want to learn more about Keras functional API : [1],[2]
The encoder is made up of :
Input Layer : Takes the English sentence and pass it to the embedding layer.Embedding Layer : Takes the English sentence and convert each word to fixed size vectorFirst LSTM Layer : Every time step, it takes a vector that represents a word and pass its output to the next layer, We used CuDNNLSTM layer instead of LSTM because it’s much much faster.Second LSTM Layer : It does the same thing as the previous layer, but instead of passing its output, it passes its states to the first LSTM layer of the decoder .
Input Layer : Takes the English sentence and pass it to the embedding layer.
Embedding Layer : Takes the English sentence and convert each word to fixed size vector
First LSTM Layer : Every time step, it takes a vector that represents a word and pass its output to the next layer, We used CuDNNLSTM layer instead of LSTM because it’s much much faster.
Second LSTM Layer : It does the same thing as the previous layer, but instead of passing its output, it passes its states to the first LSTM layer of the decoder .
Building the decoder :
The decoder is made up of :
Input Layer : Takes the French sentence and pass it to the embedding layer.Embedding Layer : Takes the French sentence and convert each word to fixed size vectorFirst LSTM Layer : Every time step, it takes a vector that represents a word and pass its output to the next layer, but here in the decoder, we initialize the state of this layer to be the last state of the last LSTM layer from the decoder .Second LSTM Layer : Processing the output from the previous layer and passes its output to a dense layer .Dense Layer (Output Layer) : Takes the output from the previous layer and outputs a one hot vector representing the target French word
Input Layer : Takes the French sentence and pass it to the embedding layer.
Embedding Layer : Takes the French sentence and convert each word to fixed size vector
First LSTM Layer : Every time step, it takes a vector that represents a word and pass its output to the next layer, but here in the decoder, we initialize the state of this layer to be the last state of the last LSTM layer from the decoder .
Second LSTM Layer : Processing the output from the previous layer and passes its output to a dense layer .
Dense Layer (Output Layer) : Takes the output from the previous layer and outputs a one hot vector representing the target French word
Note :
We have to know that we don’t convert each English sentence into French in one time step, we do that in a number of time steps that equals the number of words that the longest English sentence has.
so if the longest English sentence has 10 words, we have to take 10 time steps to get its French translation.
I simplified the encoder decoder model using the previous blocks, Where in reality, we have many time steps, not just one as in the simplified representation, and the final output is not a sentence that we can read directly, what is really happening is that every time step we get a one hot vector that represent the target word.
Bringing the encoder and decoder together into one model :
I only used 9000 samples because I did not have enough memory to use the whole dataset, and because of that, the result wasn’t good enough.
The optimizer we used is rmsprop , it is used in models that have recurrent layers.
This optimizer is usually a good choice for recurrent neural networks.
We used categorical_crossentropy as the loss function, because we treat each word as a category (our output is a one hot vector that represents a word)
We got to know what an encoder decoder model is.
How to create the encoder
How to use the embedding layer
How to make the decoder
We understood how the encoder decoder model works
We trained the model
In the next part of this series, we will use the trained model to translate English sentences to French.
1- NLP | Sequence to Sequence Networks| Part 1| Processing text data
2- NLP | Sequence to Sequence Networks| Part 2|Seq2seq Model (Encoder Decoder Model)
All the references o this series will be at the end of the last part.
You can follow me on Twitter @ModMaamari
AI Generates Taylor Swift’s Song Lyrics
Deep Neural Networks for Regression Problems
Machine Learning Crash Course with TensorFlow APIs Summary
How To Make A CNN Using Tensorflow and Keras
How to Choose the Best Machine Learning Model ?