Neural Machine Translation with Python | by Susan Li | Towards Data Science

Machine translation, sometimes referred to by the abbreviation MT is a very challenge task that investigates the use of software to translate text or speech from one language to another. Traditionally, it involves large statistical models developed using highly sophisticated linguistic knowledge.
Here we are, we are going to use deep neural networks for the problem of machine translation. We will discover how to develop a neural machine translation model for translating English to French. Our model will accept English text as input and return the French translation. To be more precise, we will be practicing building 4 models, which are:
A simple RNN.
An RNN with embedding.
A bidirectional RNN.
An encoder-decoder model.
Training and evaluating deep neural networks is a computationally intensive task. I used AWS EC2 instance to run all of the code. If you plan to follow along, you should have access to GPU instances.
import collectionsimport helperimport numpy as npimport project_tests as testsfrom keras.preprocessing.text import Tokenizerfrom keras.preprocessing.sequence import pad_sequencesfrom keras.models import Modelfrom keras.layers import GRU, Input, Dense, TimeDistributed, Activation, RepeatVector, Bidirectionalfrom keras.layers.embeddings import Embeddingfrom keras.optimizers import Adamfrom keras.losses import sparse_categorical_crossentropy
I use help.pyto load the data, and project_test.pyis for testing our functions.
The dataset contains a relative small vocabulary and can be found here. The small_vocab_en file contains English sentences and their French translations in the small_vocab_fr file.
Load the data
english_sentences = helper.load_data('data/small_vocab_en')french_sentences = helper.load_data('data/small_vocab_fr')print('Dataset Loaded')
Dataset Loaded
Sample sentences
Each line in small_vocab_en contains an English sentence with the respective translation in each line of small_vocab_fr.
for sample_i in range(2):    print('small_vocab_en Line {}:  {}'.format(sample_i + 1, english_sentences[sample_i]))    print('small_vocab_fr Line {}:  {}'.format(sample_i + 1, french_sentences[sample_i]))
Vocabulary
The complexity of the problem is determined by the complexity of the vocabulary. A more complex vocabulary is a more complex problem. Let’s look at the complexity of the data set we’ll be working with.
english_words_counter = collections.Counter([word for sentence in english_sentences for word in sentence.split()])french_words_counter = collections.Counter([word for sentence in french_sentences for word in sentence.split()])print('{} English words.'.format(len([word for sentence in english_sentences for word in sentence.split()])))print('{} unique English words.'.format(len(english_words_counter)))print('10 Most common words in the English dataset:')print('"' + '" "'.join(list(zip(*english_words_counter.most_common(10)))[0]) + '"')print()print('{} French words.'.format(len([word for sentence in french_sentences for word in sentence.split()])))print('{} unique French words.'.format(len(french_words_counter)))print('10 Most common words in the French dataset:')print('"' + '" "'.join(list(zip(*french_words_counter.most_common(10)))[0]) + '"')
We will convert the text into sequences of integers using the following pre-process methods:
Tokenize the words into idsAdd padding to make all the sequences the same length.
Tokenize the words into ids
Add padding to make all the sequences the same length.
Tokenize
Turn each sentence into a sequence of words ids using Keras’s Tokenizer function. Use this function to tokenize english_sentences and french_sentences .
The function tokenize returns tokenized input and the tokenized class.
def tokenize(x):    x_tk = Tokenizer(char_level = False)    x_tk.fit_on_texts(x)    return x_tk.texts_to_sequences(x), x_tktext_sentences = [    'The quick brown fox jumps over the lazy dog .',    'By Jove , my quick study of lexicography won a prize .',    'This is a short sentence .']text_tokenized, text_tokenizer = tokenize(text_sentences)print(text_tokenizer.word_index)print()for sample_i, (sent, token_sent) in enumerate(zip(text_sentences, text_tokenized)):    print('Sequence {} in x'.format(sample_i + 1))    print('  Input:  {}'.format(sent))    print('  Output: {}'.format(token_sent))
Padding
Make sure all the English sequences have the same length and all the French sequences have the same length by adding padding to the end of each sequence using Keras’s pad_sequences function.
def pad(x, length=None):    if length is None:        length = max([len(sentence) for sentence in x])    return pad_sequences(x, maxlen = length, padding = 'post')tests.test_pad(pad)# Pad Tokenized outputtest_pad = pad(text_tokenized)for sample_i, (token_sent, pad_sent) in enumerate(zip(text_tokenized, test_pad)):    print('Sequence {} in x'.format(sample_i + 1))    print('  Input:  {}'.format(np.array(token_sent)))    print('  Output: {}'.format(pad_sent))
Pre-process Pipeline
Implement a pre-process function
def preprocess(x, y):    preprocess_x, x_tk = tokenize(x)    preprocess_y, y_tk = tokenize(y)preprocess_x = pad(preprocess_x)    preprocess_y = pad(preprocess_y)# Keras's sparse_categorical_crossentropy function requires the labels to be in 3 dimensions    preprocess_y = preprocess_y.reshape(*preprocess_y.shape, 1)return preprocess_x, preprocess_y, x_tk, y_tkpreproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer =\    preprocess(english_sentences, french_sentences)    max_english_sequence_length = preproc_english_sentences.shape[1]max_french_sequence_length = preproc_french_sentences.shape[1]english_vocab_size = len(english_tokenizer.word_index)french_vocab_size = len(french_tokenizer.word_index)print('Data Preprocessed')print("Max English sentence length:", max_english_sequence_length)print("Max French sentence length:", max_french_sequence_length)print("English vocabulary size:", english_vocab_size)print("French vocabulary size:", french_vocab_size)
In this section, we will experiment with various neural network architectures. We will begin by training four relatively simple architectures.
Model 1 is a simple RNN
Model 2 is a RNN with Embedding
Model 3 is a Bidirectional RNN
Model 4 is an Encoder-Decoder RNN
After experimenting with the four simple architectures, we will construct with a deeper model that designed to outperform all four models.
Ids Back to Text
The neural network will be translating the input to words ids, which isn’t the final form we want. We want the French translation. The function logits_to_textwill bridge the gab between the logits from the neural network to the French translation. We will use this function to better understand the output of the neural network.
def logits_to_text(logits, tokenizer):    index_to_words = {id: word for word, id in tokenizer.word_index.items()}    index_to_words[0] = '<PAD>'return ' '.join([index_to_words[prediction] for prediction in np.argmax(logits, 1)])print('`logits_to_text` function loaded.')
`logits_to_text` function loaded.
Model 1: RNN
We are creating a basic RNN model which is a good baseline for sequence data that translate English to French.
def simple_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):    learning_rate = 1e-3    input_seq = Input(input_shape[1:])    rnn = GRU(64, return_sequences = True)(input_seq)    logits = TimeDistributed(Dense(french_vocab_size))(rnn)    model = Model(input_seq, Activation('softmax')(logits))    model.compile(loss = sparse_categorical_crossentropy,                  optimizer = Adam(learning_rate),                  metrics = ['accuracy'])        return modeltests.test_simple_model(simple_model)tmp_x = pad(preproc_english_sentences, max_french_sequence_length)tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))# Train the neural networksimple_rnn_model = simple_model(    tmp_x.shape,    max_french_sequence_length,    english_vocab_size,    french_vocab_size)simple_rnn_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)# Print prediction(s)print(logits_to_text(simple_rnn_model.predict(tmp_x[:1])[0], french_tokenizer))
The basic RNN model’s validation accuracy ends at 0.6039.
Model 2: Embedding
An embedding is a vector representation of the word that is close to similar words in n-dimensional space, where the n represents the size of the embedding vectors. We will create a RNN model using embedding.
from keras.models import Sequentialdef embed_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):    learning_rate = 1e-3    rnn = GRU(64, return_sequences=True, activation="tanh")        embedding = Embedding(french_vocab_size, 64, input_length=input_shape[1])     logits = TimeDistributed(Dense(french_vocab_size, activation="softmax"))        model = Sequential()    #em can only be used in first layer --> Keras Documentation    model.add(embedding)    model.add(rnn)    model.add(logits)    model.compile(loss=sparse_categorical_crossentropy,                  optimizer=Adam(learning_rate),                  metrics=['accuracy'])        return modeltests.test_embed_model(embed_model)tmp_x = pad(preproc_english_sentences, max_french_sequence_length)tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2]))embeded_model = embed_model(    tmp_x.shape,    max_french_sequence_length,    english_vocab_size,    french_vocab_size)embeded_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=10, validation_split=0.2)print(logits_to_text(embeded_model.predict(tmp_x[:1])[0], french_tokenizer))
The embedding model’s validation accuracy ends at 0.8401.
Model 3: Bidirectional RNNs
def bd_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):       learning_rate = 1e-3    model = Sequential()    model.add(Bidirectional(GRU(128, return_sequences = True, dropout = 0.1),                            input_shape = input_shape[1:]))    model.add(TimeDistributed(Dense(french_vocab_size, activation = 'softmax')))    model.compile(loss = sparse_categorical_crossentropy,                  optimizer = Adam(learning_rate),                  metrics = ['accuracy'])    return modeltests.test_bd_model(bd_model)tmp_x = pad(preproc_english_sentences, preproc_french_sentences.shape[1])tmp_x = tmp_x.reshape((-1, preproc_french_sentences.shape[-2], 1))bidi_model = bd_model(    tmp_x.shape,    preproc_french_sentences.shape[1],    len(english_tokenizer.word_index)+1,    len(french_tokenizer.word_index)+1)bidi_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=20, validation_split=0.2)# Print prediction(s)print(logits_to_text(bidi_model.predict(tmp_x[:1])[0], french_tokenizer))
The Bidirectional RNN model’s validation accuracy ends at 0.5992.
Model 4: Encoder-Decoder
The encoder creates a matrix representation of the sentence. The decoder takes this matrix as input and predicts the translation as output.
def encdec_model(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):      learning_rate = 1e-3    model = Sequential()    model.add(GRU(128, input_shape = input_shape[1:], return_sequences = False))    model.add(RepeatVector(output_sequence_length))    model.add(GRU(128, return_sequences = True))    model.add(TimeDistributed(Dense(french_vocab_size, activation = 'softmax')))        model.compile(loss = sparse_categorical_crossentropy,                  optimizer = Adam(learning_rate),                  metrics = ['accuracy'])    return modeltests.test_encdec_model(encdec_model)tmp_x = pad(preproc_english_sentences)tmp_x = tmp_x.reshape((-1, preproc_english_sentences.shape[1], 1))encodeco_model = encdec_model(    tmp_x.shape,    preproc_french_sentences.shape[1],    len(english_tokenizer.word_index)+1,    len(french_tokenizer.word_index)+1)encodeco_model.fit(tmp_x, preproc_french_sentences, batch_size=1024, epochs=20, validation_split=0.2)print(logits_to_text(encodeco_model.predict(tmp_x[:1])[0], french_tokenizer))
The Encoder-decoder model’s validation accuracy ends at 0.6406.
Model 5: Custom
Create a model_final that incorporates embedding and a bidirectional RNN into one model.
At this stage, we need to do some experiments such as changing GPU parameter to 256, changing learning rate to 0.005, training our model for more (or less than) 20 epochs etc.
def model_final(input_shape, output_sequence_length, english_vocab_size, french_vocab_size):      model = Sequential()    model.add(Embedding(input_dim=english_vocab_size,output_dim=128,input_length=input_shape[1]))    model.add(Bidirectional(GRU(256,return_sequences=False)))    model.add(RepeatVector(output_sequence_length))    model.add(Bidirectional(GRU(256,return_sequences=True)))    model.add(TimeDistributed(Dense(french_vocab_size,activation='softmax')))    learning_rate = 0.005        model.compile(loss = sparse_categorical_crossentropy,                  optimizer = Adam(learning_rate),                  metrics = ['accuracy'])        return modeltests.test_model_final(model_final)print('Final Model Loaded')
Final Model Loaded
def final_predictions(x, y, x_tk, y_tk):    tmp_X = pad(preproc_english_sentences)    model = model_final(tmp_X.shape,                        preproc_french_sentences.shape[1],                        len(english_tokenizer.word_index)+1,                        len(french_tokenizer.word_index)+1)        model.fit(tmp_X, preproc_french_sentences, batch_size = 1024, epochs = 17, validation_split = 0.2)     y_id_to_word = {value: key for key, value in y_tk.word_index.items()}    y_id_to_word[0] = '<PAD>'sentence = 'he saw a old yellow truck'    sentence = [x_tk.word_index[word] for word in sentence.split()]    sentence = pad_sequences([sentence], maxlen=x.shape[-1], padding='post')    sentences = np.array([sentence[0], x[0]])    predictions = model.predict(sentences, len(sentences))print('Sample 1:')    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[0]]))    print('Il a vu un vieux camion jaune')    print('Sample 2:')    print(' '.join([y_id_to_word[np.argmax(x)] for x in predictions[1]]))    print(' '.join([y_id_to_word[np.max(x)] for x in y[0]]))final_predictions(preproc_english_sentences, preproc_french_sentences, english_tokenizer, french_tokenizer)
We are getting perfect translations on both sentences and 0.9776 validation accuracy score!
Source code can be found at Github. I look forward to hearing feedback or questions.