Title: Neural Machine Translation - PyImageSearch

URL Source: https://pyimagesearch.com/2022/08/15/neural-machine-translation/

Published Time: 2022-08-15T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Neural Machine Translation](#h2BlogTitle)

*   [Introduction](#h3Introduction)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Having)
*   [Probabilistic Overview of Neural Machine Translation](#h3Probabilistic)
*   [Math Behind Neural Machine Translation](#h3Math)
*   [What Lies Ahead?](#h3What)
*   [Dataset](#h3Dataset)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, you will learn about the core concepts of Neural Machine Translation and a primer on attention.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20392'%3E%3C/svg%3E)

This lesson is the last in a 3-part series on **NLP 102**:

1.  [_Introduction to Recurrent Neural Networks with Keras and TensorFlow_](https://pyimg.co/a3dwm)
2.  [_Long Short-Term Memory Networks_](https://pyimg.co/wtby9)
3.  [**_Neural Machine Translation_**](https://pyimg.co/4yi97) **(today’s tutorial)**

**To learn how neural machine translation works and the math behind it, _just keep reading._**

* * *

[**Neural Machine Translation**](#TOC)
--------------------------------------

* * *

### [**Introduction**](#TOC)

Imagine you find an article about a really interesting topic on the internet. As luck might have it, it is not in your mother tongue or the language you are comfortable conversing in. You chuckle to yourself but then find an option to translate the text (as shown in **Figure 1**).

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20338'%3E%3C/svg%3E)

**Figure 1:** Translating from a source language (English) to a target language (French).

You thank the technology gods, read the article, and move on with your day. But something clicked, the webpage may have been more than 2500 words, yet the translation happened in a matter of seconds. So, unless there is a super fast person smashing keyboards somewhere inside the browser, this must have been done by an algorithm.

But how is the algorithm so accurate? What makes such an algorithm perform robustly in any language in the world?

This is a special area of Natural Language Processing known as **Neural Machine Translation**, defined as the act of translation with the help of an artificial neural network.

In this tutorial, we will learn about:

*   How Neural Machine Translation works
*   An overview of two very important papers for this task
    *   [Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)
    *   [Effective Approaches to Attention-Based Neural Machine Translation](https://arxiv.org/abs/1508.04025)
*   The dataset we will be using in later tutorials

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need to have the TensorFlow and the TensorFlow Text library installed on your system.

Luckily, both are pip-installable:

$ pip install tensorflow
$ pip install tensorflow-text

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 2:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Probabilistic Overview of Neural Machine Translation**](#TOC)

If that sounds like a mouthful, don’t be worried, we promise all of this will make sense in a minute. But, before we go any further, let us first take a moment to refresh our memory of _probability_.

![Image 4: P(Y|X)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2056%2018'%3E%3C/svg%3E) refers to a probability of the event ![Image 5: Y](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) occurring given ![Image 6: X](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2011'%3E%3C/svg%3E) has occurred. Now imagine ![Image 7: F](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) and ![Image 8: E](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2011'%3E%3C/svg%3E) as a sequence of French and English words, respectively. If we apply the same definition of conditional probability here, it will mean ![Image 9: P(F|E)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2054%2018'%3E%3C/svg%3E) is the probability of a sequence of words in French ![Image 10: (F)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2023%2018'%3E%3C/svg%3E) occurring, given there is a sequence of words in English ![Image 11: (E)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2023%2018'%3E%3C/svg%3E).

This means the task of translation (from English to French) is to maximize this probability ![Image 12: P(F|E)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2054%2018'%3E%3C/svg%3E), as shown in **Figure 3**.

![Image 13: \text{argmax}_{F}(P(F|E))](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20129%2018'%3E%3C/svg%3E)

![Image 14](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20338'%3E%3C/svg%3E)

**Figure 3:** Maximizing the conditional probability.

The neural network’s task is to learn the conditional distribution, and then when a source sentence is given, search for an appropriate target sentence by maximizing this conditional probability.

* * *

### [**Math Behind Neural Machine Translation**](#TOC)

Neural Machine Translation (NMT) is the process of leveraging an artificial neural network to maximize this conditional probability.

An NMT architecture usually comprises an encoder and a decoder, as shown in **Figure 4**.

[![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20712%20363'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/08/neural-machine-translation.png)

**Figure 4:** The encoder and decoder architecture of a Neural Machine Translation model.

Before [Bahdanau](https://arxiv.org/abs/1409.0473) and [Luong](https://arxiv.org/abs/1508.04025), the encoder and decoder used only recurrence to solve the machine translation task. In this section, we will discuss the **math** behind modeling translation using only RNNs as encoders and decoders.

Let us consider the equation of the hidden state of the RNN in the encoder.

![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 1**

Here ![Image 17: f](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2014'%3E%3C/svg%3E) is a network (can be an RNN, LSTM, or GRU). The main motivation here is to understand that the current hidden state (![Image 18: h_t](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2014'%3E%3C/svg%3E)) depends on the current input (![Image 19: x_t](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E)) and the previous hidden state (![Image 20: h_{t-1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2027%2015'%3E%3C/svg%3E)). This recursive cell output feeding to the next has already been explained in our [Introduction to RNN blog post](https://pyimg.co/a3dwm). We advise you to quickly read our RNN series (if not done already) to get a primer on the same.

The encoder in NMT creates a **bottleneck fixed-size vector** (context vector, ![Image 21: c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E)) from all the hidden states of the encoder. The context vector (![Image 22: c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E)) will be used by the decoder to get to the target sequence.

![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 2**

![Image 24: q](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E) can be any non-linearity. You will most likely find ![Image 25: c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E) to be the last hidden state ![Image 26: h_{T_{x}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2022%2015'%3E%3C/svg%3E)

The decoder predicts the next word ![Image 27: y_t](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E) given the context vector (![Image 28: c](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%207%207'%3E%3C/svg%3E)) and all the previously predicted words {![Image 29: y_1, y_2, \dots, y_{t-1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2099%2011'%3E%3C/svg%3E)}.

Now let us rewrite the probabilistic equation.

![Image 30](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 3**

![Image 31: s_t=f(y_{t-1},s_{t-1},c)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20133%2018'%3E%3C/svg%3E) is the hidden state of the decoder. Just like the hidden state of the encoder, ![Image 32: f](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%2014'%3E%3C/svg%3E) can be any recurrent architecture (RNN, LSTM, or GRU).

![Image 33: g](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2010'%3E%3C/svg%3E) can be any non-linearity that outputs the probability of the next word given all the previously generated words and the context vector.

For a translation task, we have to generate target words that maximize the conditional probability ![Image 34: p(y|x)](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2043%2018'%3E%3C/svg%3E).

![Image 35](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 4**

**TL;DR:** What essentially happens is a variable length sequence is passed to an encoder, which squishes the representation of the entire sequence into a **fixed context vector**. This context vector is then passed to the decoder, which converts it into the target sequence.

* * *

### [**What Lies Ahead?**](#TOC)

In the introduction, we mentioned two seminal papers:

*   [**Neural Machine Translation by Jointly Learning to Align and Translate**](https://arxiv.org/abs/1409.0473)**:** In this paper, the authors argue that encoding a variable-length sequence into a fixed-length context vector would deteriorate the performance of the translation.To counter the problem, they propose a soft-attention scheme. With the attention in place, the context vector would now have a complete overview of the entire input sequence.
    
    For every word translated, a dynamic context vector is built just for the current translation.
    

*   [**Effective Approaches to Attention-Based Neural Machine Translation**](https://arxiv.org/abs/1508.04025)**:** In this paper, the authors improve several key factors of its predecessor ([Neural Machine Translation by Jointly Learning to Align and Translate](https://arxiv.org/abs/1409.0473)).These include introducing unidirectional RNN for the encoder and multiplicative addition in place of additive addition. This paper aimed to build a better and more effective (as the name suggests) approach toward attention-based Neural Machine Translation.
    

* * *

### [**Dataset**](#TOC)

Are you wondering how to code this up and have your own translation algorithm?

We will need this dataset in two upcoming blog posts where we discuss the Bahdanau and Luong attentions.

Since this is a text translation task, we will need a text pair for this task. We are using the French to English dataset from [http://www.manythings.org/anki/](http://www.manythings.org/anki/)

You can download the dataset in a Colab Notebook or your local system using the following code snippet:

$ wget https://www.manythings.org/anki/fra-eng.zip
$ unzip fra-eng.zip
$ rm \_about.txt fra-eng.zip

We will expand on loading and processing the dataset in upcoming tutorials.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

This tutorial introduces Neural Machine Translation. We learn how Neural Machine Translation can be expressed in Probabilistic terms. We saw how NMT architectures are usually designed in the field.

Next, we will learn about the Bahdanau and Luong attentions and their code implementations in TensorFlow and Keras.

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “Neural Machine Translation,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, [https://pyimg.co/4yi97](https://pyimg.co/4yi97)

@incollection{ADR\_2022\_NMT,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Neural Machine Translation},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/4yi97},
}

* * *

![Image 36: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

![Image 37](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.