Neural Machine Translation (NMT) with Attention Mechanism | by Harshil Patel | Towards Data Science

It is an undeniable truth that in this era of globalization, language translation plays a vital role in communication among the denizens of different nation’s. Moreover, in a country like India - which a multilingual nation, language difference could be observed in its states itself! Hence, considering the importance of Language Translation, it becomes necessary to develop a system that could easily translate an unknown language to a known language.
In accordance with this, in this story, we will make a Deep Learning model that will translate English sentences to Marathi sentences. I have chosen Marathi language as it is easy for me to recognize. You can use any other language that is comfortable for you, as the model nearly remains the same. Moreover, I will try to briefly explain a major concept in language processing called Attention Mechanism here!
Working of Long Short Time Memory(LSTM) cellsWorking of TensorFlow, Keras and some other mandatory python libraries.
Working of Long Short Time Memory(LSTM) cells
Working of TensorFlow, Keras and some other mandatory python libraries.
The major drawback of encoder-decoder model in sequence to sequence recurrent neural network is that it can only work on short sequences. It is difficult for the encoder model to memorize long sequences and convert it into a fixed-length vector. Moreover, the decoder receives only one information that is the last encoder hidden state. Hence it's difficult for the decoder to summarize large input sequence at once. So, how do we overcome this problem?
How about if we give a vector representation from every encoder step to the decoder model!
Now, this is where the concept of ‘Attention Mechanism’ comes. The major intuition about this is that it predicts the next word by concentrating on a few relevant parts of the sequence rather than looking on the entire sequence.
In layman terms it can be described as an interference between encoder and decoder which extracts useful information from encoder and transmits it back to the decoder.
Refer here for detailed understanding about Attention Mechanism.
There are mainly two types of attention mechanism:
Global Attention
Local Attention
Global Attention are those attention in which all the hidden state vectors of the encoder are passed to get the context vector.
Local Attention are those attention in which only a few hidden state vectors of encoder are considered for the generation of context vector.
We will be using global attention in this story. Let’s now make use of attention mechanism and develop a language translator that will convert English sentence to Marathi sentence.
Open Jupyter Notebook and import some required libraries:
import pandas as pdfrom sklearn.model_selection import train_test_splitimport stringfrom string import digitsimport refrom sklearn.utils import shufflefrom tensorflow.keras.preprocessing.sequence import pad_sequencesfrom tensorflow.keras.layers import LSTM, Input, Dense,Embedding, Concatenate, TimeDistributedfrom tensorflow.keras.models import Model,load_model, model_from_jsonfrom tensorflow.keras.utils import plot_modelfrom tensorflow.keras.preprocessing.text import one_hot, Tokenizerfrom tensorflow.keras.callbacks import EarlyStoppingimport pickle as pklimport numpy as np
We will be working on a language dataset available here.
This site contains a dataset of numerous languages and its translation to English. You can download any language dataset as per your preference and comfort. However, remember to choose a dataset which is quite huge, so that we can get better results after training the model. Here, I will be downloading Marathi-English dataset which comprises of 38696 sentences.
After downloading load dataset, import the data as mentioned below:
with open('mar.txt','r') as f:  data = f.read()
Data Transformation
As you can see that it is a raw text file and hence it is necessary to clean and transform it as per our preference. We will separate Marathi and English sentences and form a list of it, continuing it by storing it into a dataframe so that it's easy for us to reuse it again easily.
uncleaned_data_list = data.split('\n')len(uncleaned_data_list)uncleaned_data_list = uncleaned_data_list[:38695]len(uncleaned_data_list)english_word = []marathi_word = []cleaned_data_list = []for word in uncleaned_data_list:  english_word.append(word.split('\t')[:-1][0])  marathi_word.append(word.split('\t')[:-1][1])language_data = pd.DataFrame(columns=['English','Marathi'])language_data['English'] = english_wordlanguage_data['Marathi'] = marathi_wordlanguage_data.to_csv('language_data.csv', index=False)
language_data.head()
english_text = language_data['English'].valuesmarathi_text = language_data['Marathi'].valueslen(english_text), len(marathi_text)
Data Cleaning
Now let’s clean the data and make it suitable for our model. In the cleaning process, we will convert into lower case, remove all punctuation and other unnecessary letters and digits too.
#to lower caseenglish_text_ = [x.lower() for x in english_text]marathi_text_ = [x.lower() for x in marathi_text]#removing inverted commasenglish_text_ = [re.sub("'",'',x) for x in english_text_]marathi_text_ = [re.sub("'",'',x) for x in marathi_text_]def remove_punc(text_list):  table = str.maketrans('', '', string.punctuation)  removed_punc_text = []  for sent in text_list:    sentance = [w.translate(table) for w in sent.split(' ')]    removed_punc_text.append(' '.join(sentance))  return removed_punc_textenglish_text_ = remove_punc(english_text_)marathi_text_ = remove_punc(marathi_text_)remove_digits = str.maketrans('', '', digits)removed_digits_text = []for sent in english_text_:  sentance = [w.translate(remove_digits) for w in sent.split(' ')]  removed_digits_text.append(' '.join(sentance))english_text_ = removed_digits_text# removing the digits from the marathi sentancesmarathi_text_ = [re.sub("[२३०८१५७९४६]","",x) for x in marathi_text_]marathi_text_ = [re.sub("[\u200d]","",x) for x in marathi_text_]# removing the stating and ending whitespacesenglish_text_ = [x.strip() for x in english_text_]marathi_text_ = [x.strip() for x in marathi_text_]
Adding ‘start’ and ‘end’ tag to marathi sentence. This will help the decoder to know from where to start decoding and when to end.
# Putting the start and end words in the marathi sentancesmarathi_text_ = ["start " + x + " end" for x in marathi_text_]# manipulated_marathi_text_marathi_text_[0], english_text_[0]
(‘start जा end’, ‘go’)
We will split our dataset with a ratio of 0.1 so that our trained model can give precise results. X_train and y_train will be our training set while X_test and y_test will be our testing/validation set.
X = english_text_Y = marathi_text_X_train, X_test, y_train, y_test=train_test_split(X,Y,test_size=0.1)
Let’s determine the maximum length of our sentences in both English and Marathi:
def Max_length(data):  max_length_ = max([len(x.split(' ')) for x in data])  return max_length_#Training datamax_length_english = Max_length(X_train)max_length_marathi = Max_length(y_train)#Test datamax_length_english_test = Max_length(X_test)max_length_marathi_test = Max_length(y_test)max_length_marathi, max_length_english
(26, 32)
Tokenization:
As a neural network requires numerical data to process, it becomes necessary to convert our string input to a numerical list. One way of doing this is to use Tokenizer provided by keras-preprocessing library.
Also, remember it is mandatory to have an equal length of all input sequences in sequence-to-sequence models. So, we will pad extra ‘0s’ to make the sequence of the same length. This would be done by pad_sequence.
englishTokenizer = Tokenizer()englishTokenizer.fit_on_texts(X_train)Eword2index = englishTokenizer.word_indexvocab_size_source = len(Eword2index) + 1X_train = englishTokenizer.texts_to_sequences(X_train)X_train = pad_sequences(X_train, maxlen=max_length_english, padding='post')X_test = englishTokenizer.texts_to_sequences(X_test)X_test = pad_sequences(X_test, maxlen = max_length_english, padding='post')marathiTokenizer = Tokenizer()marathiTokenizer.fit_on_texts(y_train)Mword2index = marathiTokenizer.word_indexvocab_size_target = len(Mword2index) + 1y_train = marathiTokenizer.texts_to_sequences(y_train)y_train = pad_sequences(y_train, maxlen=max_length_marathi, padding='post')y_test = marathiTokenizer.texts_to_sequences(y_test)y_test = pad_sequences(y_test, maxlen = max_length_marathi, padding='post')vocab_size_source, vocab_size_target
(5413, 12789)
X_train[0], y_train[0]
(array([ 1, 157, 5, 134, 4, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32), array([ 1, 6, 22, 61, 253, 29, 2, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0], dtype=int32))
To save our preprocessing time whenever we reuse it again in future, we will save our important attributes. So, let’s do it first with the help of pickle library.
with open('NMT_data.pkl','wb') as f:  pkl.dump([X_train, y_train, X_test, y_test],f)with open('NMT_Etokenizer.pkl','wb') as f:  pkl.dump([vocab_size_source, Eword2index, englishTokenizer], f)with open('NMT_Mtokenizer.pkl', 'wb') as f:  pkl.dump([vocab_size_target, Mword2index, marathiTokenizer], f)X_train = np.array(X_train)y_train = np.array(y_train)X_test = np.array(X_test)y_test = np.array(y_test)
Instead of a simple encoder-decoder architecture, we will be using Attention Mechanism as discussed earlier in this blog.
Keras does not officially support attention layer. So, we can either implement our own attention layer or use a third-party implementation. For now, we will be using a third party attention mechanism. You can download the attention layer from here and copy it in a different file called attention.py. This attention is an implementation of ‘Bahdanau Attention’ .
Let’s define the structure of our model:
from attention import AttentionLayerfrom keras import backend as K K.clear_session() latent_dim = 500# Encoder encoder_inputs = Input(shape=(max_length_english,)) enc_emb = Embedding(vocab_size_source, latent_dim,trainable=True)(encoder_inputs)#LSTM 1 encoder_lstm1 = LSTM(latent_dim,return_sequences=True,return_state=True) encoder_output1, state_h1, state_c1 = encoder_lstm1(enc_emb)#LSTM 2 encoder_lstm2 = LSTM(latent_dim,return_sequences=True,return_state=True) encoder_output2, state_h2, state_c2 = encoder_lstm2(encoder_output1)#LSTM 3 encoder_lstm3=LSTM(latent_dim, return_state=True, return_sequences=True) encoder_outputs, state_h, state_c= encoder_lstm3(encoder_output2)# Set up the decoder. decoder_inputs = Input(shape=(None,)) dec_emb_layer = Embedding(vocab_size_target, latent_dim,trainable=True) dec_emb = dec_emb_layer(decoder_inputs)#LSTM using encoder_states as initial statedecoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True) decoder_outputs,decoder_fwd_state, decoder_back_state = decoder_lstm(dec_emb,initial_state=[state_h, state_c])#Attention Layerattn_layer = AttentionLayer(name='attention_layer') attn_out, attn_states = attn_layer([encoder_outputs, decoder_outputs])# Concat attention output and decoder LSTM output decoder_concat_input = Concatenate(axis=-1, name='concat_layer')([decoder_outputs, attn_out])#Dense layerdecoder_dense = TimeDistributed(Dense(vocab_size_target, activation='softmax')) decoder_outputs = decoder_dense(decoder_concat_input)# Define the modelmodel = Model([encoder_inputs, decoder_inputs], decoder_outputs) plot_model(model, to_file='train_model.png', show_shapes=True)
You can modify this model as per your choice and requirement to get better results. You can change number of layers, number of units or some regularization techniques too. For the time being, let’s move forward and see what our model looks like!
model.compile(optimizer='rmsprop',              loss='sparse_categorical_crossentropy',               metrics=['accuracy'])
We will first define some callbacks so that it would be easy for model visualization and evaluation in future.
es = EarlyStopping(monitor='val_loss', mode='min', verbose=1)
We are using ‘Teacher Forcing’ technique for faster training of our model. In the teacher forcing method, we also pass the target data as the input to the decoder. For example, if we are going to predict ‘hello’, then we will pass ‘hello’ itself as an input to the decoder. Due to this it makes the learning process faster.
Let's train our Model:
history = model.fit([X_train, y_train[:,:-1]], y_train.reshape(y_train.shape[0], y_train.shape[1],1)[:,1:],                     epochs=50,                     callbacks=[es],                    batch_size=512,                    validation_data = ([X_test, y_test[:,:-1]],           y_test.reshape(y_test.shape[0], y_test.shape[1], 1)[:,1:]))
The execution time was around 39 seconds per epoch on 12GB NVIDIA Tesla K80 GPU. EarlyStopping was achieved at 18th epoch.
We can visualize the loss difference in both training and validation phase as:
from matplotlib import pyplot pyplot.plot(history.history['loss'], label='train') pyplot.plot(history.history['val_loss'], label='test') pyplot.legend() pyplot.show()
We are getting some pretty good results from our model with around 90% validation accuracy and a validation loss of 0.5303.
Let’s save our trained model with proper weights. Do remember to save the model like I have done as we have to load weights too for the inference model.
model_json = model.to_json()with open("NMT_model.json", "w") as json_file:    json_file.write(model_json)# serialize weights to HDF5model.save_weights("NMT_model_weight.h5")print("Saved model to disk")
Load model:
# loading the model architecture and asigning the weightsjson_file = open('NMT_model.json', 'r')loaded_model_json = json_file.read()json_file.close()model_loaded = model_from_json(loaded_model_json, custom_objects={'AttentionLayer': AttentionLayer})# load weights into new modelmodel_loaded.load_weights("NMT_model_weight.h5")
In machine learning we use inference model to predict our output sequences by considering weights from a pre-trained model. In other terms, it can be said that its a model that deduces properties that are learned in training phase and are now used for predicting new sequences.
Let’s code our inference model:
latent_dim=500# encoder inferenceencoder_inputs = model_loaded.input[0]  #loading encoder_inputsencoder_outputs, state_h, state_c = model_loaded.layers[6].output #loading encoder_outputs#print(encoder_outputs.shape)encoder_model = Model(inputs=encoder_inputs,outputs=[encoder_outputs, state_h, state_c])# decoder inference# Below tensors will hold the states of the previous time stepdecoder_state_input_h = Input(shape=(latent_dim,))decoder_state_input_c = Input(shape=(latent_dim,))decoder_hidden_state_input = Input(shape=(32,latent_dim))# Get the embeddings of the decoder sequencedecoder_inputs = model_loaded.layers[3].output#print(decoder_inputs.shape)dec_emb_layer = model_loaded.layers[5]dec_emb2= dec_emb_layer(decoder_inputs)# To predict the next word in the sequence, set the initial states to the states from the previous time stepdecoder_lstm = model_loaded.layers[7]decoder_outputs2, state_h2, state_c2 = decoder_lstm(dec_emb2, initial_state=[decoder_state_input_h, decoder_state_input_c])#attention inferenceattn_layer = model_loaded.layers[8]attn_out_inf, attn_states_inf = attn_layer([decoder_hidden_state_input, decoder_outputs2])concate = model_loaded.layers[9]decoder_inf_concat = concate([decoder_outputs2, attn_out_inf])# A dense softmax layer to generate prob dist. over the target vocabularydecoder_dense = model_loaded.layers[10]decoder_outputs2 = decoder_dense(decoder_inf_concat)# Final decoder modeldecoder_model = Model([decoder_inputs] + [decoder_hidden_state_input,decoder_state_input_h, decoder_state_input_c],[decoder_outputs2] + [state_h2, state_c2])
Now we have trained the sequence to sequence model and created the inference model using the trained model for making a prediction. Let’s predict some Marathi sentences from the English sentences.
def decode_sequence(input_seq):    # Encode the input as state vectors.    e_out, e_h, e_c = encoder_model.predict(input_seq)# Generate empty target sequence of length 1.    target_seq = np.zeros((1,1))# Chose the 'start' word as the first word of the target sequence    target_seq[0, 0] = Mword2index['start']stop_condition = False    decoded_sentence = ''    while not stop_condition:        output_tokens, h, c = decoder_model.predict([target_seq] + [e_out, e_h, e_c])# Sample a token        sampled_token_index = np.argmax(output_tokens[0, -1, :])        if sampled_token_index == 0:          break        else:          sampled_token = Mindex2word[sampled_token_index]if(sampled_token!='end'):              decoded_sentence += ' '+sampled_token# Exit condition: either hit max length or find stop word.              if (sampled_token == 'end' or len(decoded_sentence.split()) >= (26-1)):                  stop_condition = True# Update the target sequence (of length 1).          target_seq = np.zeros((1,1))          target_seq[0, 0] = sampled_token_index# Update internal states          e_h, e_c = h, creturn decoded_sentence
Forming a reverse vocabulary:
Eindex2word = englishTokenizer.index_wordMindex2word = marathiTokenizer.index_word
Some transformation before giving a string to the function:
def seq2summary(input_seq):    newString=''    for i in input_seq:      if((i!=0 and i!=Mword2index['start']) and i!=Mword2index['end']):        newString=newString+Mindex2word[i]+' '    return newStringdef seq2text(input_seq):    newString=''    for i in input_seq:      if(i!=0):        newString=newString+Eindex2word[i]+' '    return newString
Call the necessary functions and let’s test our translation model:
for i in range(10):    print("Review:",seq2text(X_test[i]))  print("Original summary:",seq2summary(y_test[i]))  print("Predicted summary:",decode_sequence(X_test[i].reshape(1,32)))  print("\n")
Review: no one will tell you Original summary: तुला कोणीही सांगणार नाही Predicted summary: कोणीही तुला सांगणार नाही
Review: look ahead Original summary: समोर बघा Predicted summary: तिथे बघ
Review: im going to return this to tom Original summary: मी हे टॉमला परत करायला जातोय Predicted summary: मी ते स्वतःहून करणार आहे
Review: an eagle is flying in the sky Original summary: आकाशात एक गरुड आहे Predicted summary: न्यूयॉर्क अतिरेकी तो दुसर्याचा
Review: he speaks arabic Original summary: तो अरबी बोलतो Predicted summary: तो अरबी बोलतो
Review: clean up this mess Original summary: हा पसारा साफ करून टाका Predicted summary: हा पसारा साफ कर
Review: dont speak french in the class Original summary: वर्गात फ्रेंचमध्ये बोलू नका Predicted summary: वर्गात जास्त कठीण राहू नकोस
Review: i turned the lights out Original summary: मी दिवे Predicted summary: मी दोन हात वर केला
Review: how many rackets do you have Original summary: तुझ्याकडे किती रॅकेट आहेत Predicted summary: तुमच्याकडे किती बहिणी आहेत
Review: i gave tom marys phone number Original summary: मी टॉमला मेरीचा फोन नंबर दिला Predicted summary: मी टॉमला मेरीचा फोन क्रमांक दिला
Hurrah!!
Our model makes some good translations of English sentences to Marathi sentences.
I have deployed my model through Django and have hosted it on heroku. You can look at it here.
hurdlenet.herokuapp.com
In this story, we learned about the functionality of Attention Mechanism and implemented a Language Translation task. This task could have multiple use cases in daily lifestyles. For example, we can use this technique to build a multi-language translator that can translate various languages from a single language. Also, if we can integrate this with an Optical Character Recognition system, we can translate texts directly from images.
If you have any other use case or technique to work with translation data and also, if you find a more improved model for NMT, do share in the response block below!
The entire code for this article is available here. If you have any feedback, feel free to reach out to me on LinkedIn.