Implementing neural machine translation using keras | by Renu Khandelwal | Towards Data Science

In this article, you get a step by step explanation of building a neural machine translator using English as the source language and Spanish as the target language. We will be using Long Short Term Memory(LSTM) units in keras
towardsdatascience.com
In this article, we will create a Sequence to Sequence model using an encoder and decoder framework with LSTM as the basic block
Sequence to Sequence model maps a source sequence to target sequence. The source sequence is input language to the machine translation system, and the target sequence is the output language.
A step by step implementation of a neural machine translation(NMT) using Teacher forcing without Attention mechanism. Implementation is using keras library with LSTM as the basic block
Reading the data from the file containing the source and target sentences
Cleaning the data by converting to lowercase, removing spaces, special characters, digits, and quotes
Tag the start and end of the target sentence using START_ and _END respectively for training and inference
Create the dictionary of unique source and target words to vector and vice-versa
Shuffle the data for better generalization
Split the dataset into train and test data
Create the data; we will be using fit_generator() to fit the data to the model
Build the encoder using Embedding and LSTM layers
Build the decoder using Embedding and LSTM layers and takes input from the embedding layer and the encoder states.
Compile the model and train the model
Make predictions from the model
Import the required libraries
import pandas as pdimport numpy as npimport stringfrom string import digitsimport matplotlib.pyplot as plt%matplotlib inlineimport refrom sklearn.utils import shufflefrom sklearn.model_selection import train_test_splitfrom keras.layers import Input, LSTM, Embedding, Densefrom keras.models import Model
Read the file that contains the English-Spanish translations that we downloaded from the here
# Path to the data txt file on disk.data_path = "\\NMT\\spa-eng\\spa.txt"# open the file eng-spa.txt and readlines= pd.read_table(data_path,  names =['source', 'target', 'comments'])#printing sample data from lineslines.sample(6)
We apply the following text cleaning
Convert text to lower case
Remove quotes
Remove all special characters like “@, !, *, $, #, ?, %, etc.”
Clean digits from the source and target sentences. If the source or the target language use different symbols for the numbers, then remove those symbols
Remove spaces
# convert source and target text to Lowercase lines.source=lines.source.apply(lambda x: x.lower())lines.target=lines.target.apply(lambda x: x.lower())# Remove quotes from source and target textlines.source=lines.source.apply(lambda x: re.sub("'", '', x))lines.target=lines.target.apply(lambda x: re.sub("'", '', x))# create a set of all special charactersspecial_characters= set(string.punctuation)# Remove all the special characterslines.source = lines.source.apply(lambda x: ''.join(char1 for char1 in x if char1 not in special_characters))lines.target = lines.target.apply(lambda x: ''.join(char1 for char1 in x if char1 not in special_characters))# Remove digits from source and target sentencesnum_digits= str.maketrans('','', digits)lines.source=lines.source.apply(lambda x: x.translate(num_digits))lines.target= lines.target.apply(lambda x: x.translate(num_digits))# Remove extra spaceslines.source=lines.source.apply(lambda x: x.strip())lines.target=lines.target.apply(lambda x: x.strip())lines.source=lines.source.apply(lambda x: re.sub(" +", " ", x))lines.target=lines.target.apply(lambda x: re.sub(" +", " ", x))
Adding the START_ and the _END token to the target sentences is very useful for training and during inference. These tags help to know when to start the translation and when to end the translation.
# Add start and end tokens to target sequenceslines.target = lines.target.apply(lambda x : 'START_ '+ x + ' _END')lines.sample(6)
START_ tag marks the start of the target sentences, and an _END tag marks the completion of the target sentence.
Create a set of unique words both for source and target language from the dataset and sort them alphabetically
# Find all the source and target words and sort them# Vocabulary of Source languageall_source_words=set()for source in lines.source:    for word in source.split():        if word not in all_source_words:            all_source_words.add(word)# Vocabulary of Target all_target_words=set()for target in lines.target:    for word in target.split():        if word not in all_target_words:            all_target_words.add(word)# sort all unique source and target wordssource_words= sorted(list(all_source_words))target_words=sorted(list(all_target_words))
Find the maximum length of the source and target sentences in the dataset
#Find maximum sentence length in  the source and target datasource_length_list=[]for l in lines.source:    source_length_list.append(len(l.split(' ')))max_source_length= max(source_length_list)print(" Max length of the source sentence",max_source_length)target_length_list=[]for l in lines.target:    target_length_list.append(len(l.split(' ')))max_target_length= max(target_length_list)print(" Max length of the target sentence",max_target_length)
Create a word to index dictionary and an index to word dictionary for all unique source and target words in the dataset.
Size of the word to vector will be based on the length of the source and target vocabulary
# creating a word to index(word2idx) for source and targetsource_word2idx= dict([(word, i+1) for i,word in enumerate(source_words)])target_word2idx=dict([(word, i+1) for i, word in enumerate(target_words)])
#creating a dictionary for index to word for source and target vocabularysource_idx2word= dict([(i, word) for word, i in  source_word2idx.items()])print(source_idx2word)target_idx2word =dict([(i, word) for word, i in target_word2idx.items()])
Shuffle the data
Shuffling helps with
Reducing variance
Ensures models remain generic and overfit less
Batches between epochs do not look alike
Makes model more robust
#Shuffle the datalines = shuffle(lines)
# Train - Test SplitX, y = lines.source, lines.targetX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.1)X_train.shape, X_test.shape
We will use fit_generator() instead of the fit() method as our data is too large to fit into the memory. fit_generator() needs an underlying function to generate the data.
We create the underlying function generate_batch() for generating data in batches
The fit_generator() will accept a batch of data from the underlying function, generate_batch()
To train a sequence to sequence model, we need to create one-hot encoded data for
encoder inputs: The 2D array will be of shape (batch_size, max source sentence length). For a batch_size of 128 and a max source sentence length of 47, the shape of encoder_input will be (128,47)
decoder inputs: The 2D array will be of shape (batch_size, max target sentence length). For a batch_size of 128 and a max target sentence length of 55, the shape of decoder inputs will be (128,55)
decoder outputs: The 3D array will be of shape (batch_size, max target sentence length, number of unique words in target sentences). For a batch_size of 128 and a max target sentence length of 55, the shape of decoder output will be (128,55, 27200).
Number of unique words in the target_sentence is 27199 which we zero pad, and hence the third parameter in decoder output is 27200
# Input tokens for encodernum_encoder_tokens=len(source_words)# Input tokens for decoder zero paddednum_decoder_tokens=len(target_words) +1
We now create the generator_batch function()
def generate_batch(X = X_train, y = y_train, batch_size = 128):    ''' Generate a batch of data '''    while True:        for j in range(0, len(X), batch_size):            encoder_input_data = np.zeros((batch_size, max_source_length),dtype='float32')            decoder_input_data = np.zeros((batch_size, max_target_length),dtype='float32')            decoder_target_data = np.zeros((batch_size, max_target_length, num_decoder_tokens),dtype='float32')            for i, (input_text, target_text) in enumerate(zip(X[j:j+batch_size], y[j:j+batch_size])):                for t, word in enumerate(input_text.split()):encoder_input_data[i, t] = source_word2idx[word]                 for t, word in enumerate(target_text.split()):                    if t<len(target_text.split())-1:                        decoder_input_data[i, t] = target_word2idx[word] # decoder input seq                    if t>0:                        # decoder target sequence (one hot encoded)                        # does not include the START_ token                        # Offset by one timestep                        #print(word)                        decoder_target_data[i, t - 1, target_word2idx[word]] = 1.                                yield([encoder_input_data, decoder_input_data], decoder_target_data)
We will use Teacher Forcing to train the sequence to sequence model for faster and efficient training of the decoder.
Teacher forcing algorithm trains decoder by supplying the actual output of the previous timestamp instead of the predicted output from the last time step as inputs during training.
Decoder learns to generate a word at t+1 timestep, taking into account the actual output at time step t and the encoder’s internal state; hence we offset the decoder output by one timestep.
Setup basic parameters
We set the necessary parameters like
number of training samples
number of validation samples
batch_size used for creating the training data
Epochs to train on
The latent dimension of the encoding space
train_samples = len(X_train)val_samples = len(X_test)batch_size = 128epochs = 50latent_dim=256
Build the encoder and decoder using LSTM. The encoder will encode the input sentences of the source language. Hidden state and the cell state of the encoder will be passed as input to the decoder along with actual target sequences.
The encoder will encode the input sequence. We pass the input through the input layer. The first hidden layer will be the embedding layer. Embeddings translate large sparse vectors into a dense lower-dimensional space preserving the semantic relationships.
Pass three parameters to Embedding(); the first parameter is the size of the vocabulary; the second parameter is the dimension of the dense Embedding. We set mask_zero as True as this implies that the input value of 0 is a special “padding” value that should be masked out.
Create the LSTM layer and only set return_state to True as we want to retain the hidden state and cell state of the encoder. We discard the encoder_output and preserve the hidden state and cell state of the LSTM to be passed to the decoder
# Define an input sequence and process it.encoder_inputs = Input(shape=(None,))enc_emb =  Embedding(num_encoder_tokens, latent_dim, mask_zero = True)(encoder_inputs)encoder_lstm = LSTM(latent_dim, return_state=True)encoder_outputs, state_h, state_c = encoder_lstm(enc_emb)# We discard `encoder_outputs` and only keep the states.encoder_states = [state_h, state_c]
We create an input layer for the decoder_inputs; Embedding is again the first hidden layer in the decoder.
The LSTM layer will return output sequences as well as the internal states. The internal states will be used only during the inference phase and will not be used during the training phase.
LSTM in the decoder takes input from the embedding layer and the encoder states. We apply a softmax activation to the Dense layer and then finally generate the decoder outputs
# Set up the decoder, using `encoder_states` as initial state.decoder_inputs = Input(shape=(None,))dec_emb_layer = Embedding(num_decoder_tokens, latent_dim, mask_zero = True)dec_emb = dec_emb_layer(decoder_inputs)# We set up our decoder to return full output sequences,# and to return internal states as well. We don't use the# return states in the training model, but we will use them in inference.decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)decoder_outputs, _, _ = decoder_lstm(dec_emb,                                     initial_state=encoder_states)decoder_dense = Dense(num_decoder_tokens, activation='softmax')decoder_outputs = decoder_dense(decoder_outputs)
Define the model
The sequence to sequence model takes encoder and decoder inputs to output decoder outputs
# Define the model that takes encoder and decoder input # to output decoder_outputsmodel = Model([encoder_inputs, decoder_inputs], decoder_outputs)
To train the model, we first compile the model and then fit the data to the model
We compile the model using “rmsprop” optimizer; use categorical_crossentropy as we use categorical labels which is one-hot encoded vectors
model.compile(optimizer=’rmsprop’, loss=’categorical_crossentropy’, metrics=[‘acc’])
setup the parameters to fit the model
train_samples = len(X_train) # Total Training samplesval_samples = len(X_test)    # Total validation or test samplesbatch_size = 128epochs = 100
Fit the model using fit_generator(). we had already created the underlying function for generating the data, generate_batch() for generating the training and the test dataset.
steps_per_epoch is computed by dividing the total number of training samples by the batch size. When we reach the step count, we start with a new epoch
model.fit_generator(generator = generate_batch(X_train, y_train, batch_size = batch_size),                    steps_per_epoch = train_samples//batch_size,                    epochs=epochs,                    validation_data = generate_batch(X_test, y_test, batch_size = batch_size),                    validation_steps = val_samples//batch_size)
Save the weights so that you can load them later for inference
model.save_weights(‘nmt_weights_100epochs.h5’)
Weights can be loaded from the saved weights file
model.load_weights('nmt_weights_100epochs.h5')
During inference, we want to decode unknown input sequences to predict the output.
Steps for inference
Encode the input sequences into hidden state and cell state of the LSTM
The decoder will predict one sequence at a time. The first input to the decoder will be hidden state and cell state of the encoder and the START_ tag
The output of the decoder will be fed as an input to the decoder for the next time step as shown in the diagram below
At each time step, decoder outputs one-hot encoded vector to which we apply np.argmax and convert the vector to word from the dictionary that stores index to word
Keep appending the target words generated at each time step
Repeat the steps till we hit the _END tag or the word limit
Define the inference model
# Encode the input sequence to get the "Context vectors"encoder_model = Model(encoder_inputs, encoder_states)# Decoder setup# Below tensors will hold the states of the previous time stepdecoder_state_input_h = Input(shape=(latent_dim,))decoder_state_input_c = Input(shape=(latent_dim,))decoder_state_input = [decoder_state_input_h, decoder_state_input_c]# Get the embeddings of the decoder sequencedec_emb2= dec_emb_layer(decoder_inputs)# To predict the next word in the sequence, set the initial states to the states from the previous time stepdecoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=decoder_state_input)decoder_states2 = [state_h2, state_c2]# A dense softmax layer to generate prob dist. over the target vocabularydecoder_outputs2 = decoder_dense(decoder_outputs2)# Final decoder modeldecoder_model = Model(    [decoder_inputs] + decoder_state_input,    [decoder_outputs2] + decoder_states2)
Create a function for inference lookup
def decode_sequence(input_seq):    # Encode the input as state vectors.    states_value = encoder_model.predict(input_seq)    # Generate empty target sequence of length 1.    target_seq = np.zeros((1,1))    # Populate the first character of     #target sequence with the start character.    target_seq[0, 0] = target_word2idx['START_']# Sampling loop for a batch of sequences    # (to simplify, here we assume a batch of size 1).    stop_condition = False    decoded_sentence = ''    while not stop_condition:        output_tokens, h, c = decoder_model.predict([target_seq] + states_value)# Sample a token        sampled_token_index = np.argmax(output_tokens[0, -1, :])        sampled_word =target_idx2word[sampled_token_index]        decoded_sentence += ' '+ sampled_word# Exit condition: either hit max length        # or find stop character.        if (sampled_word == '_END' or           len(decoded_sentence) > 50):            stop_condition = True# Update the target sequence (of length 1).        target_seq = np.zeros((1,1))        target_seq[0, 0] = sampled_token_index# Update states        states_value = [h, c]return decoded_sentence
train_gen = generate_batch(X_train, y_train, batch_size = 1)k=-1
Pass a source sentence and then compare the predicted output to actual output
k+=1(input_seq, actual_output), _ = next(train_gen)decoded_sentence = decode_sequence(input_seq)print(‘Input Source sentence:’, X_train[k:k+1].values[0])print(‘Actual Target Translation:’, y_train[k:k+1].values[0][6:-4])print(‘Predicted Target Translation:’, decoded_sentence[:-4])
test_gen = generate_batch(X_test, y_test, batch_size = 1)k=10k+=1(input_seq, actual_output), _ = next(test_gen)decoded_sentence = decode_sequence(input_seq)print('Input Source sentence:', X_test[k:k+1].values[0])print('Actual Target Translation:', y_test[k:k+1].values[0][6:-4])print('Predicted Target Translation:', decoded_sentence[:-4])
Some predictions were good, some reasonable and some not right.
Additional enhancements to the model
We can replace LSTM’s will GRU, add more LSTM/GRU nodes, train for more epochs and use attention mechanism