Title: Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras - PyImageSearch

URL Source: https://pyimagesearch.com/2022/08/29/neural-machine-translation-with-luongs-attention-using-tensorflow-and-keras/

Published Time: 2022-08-29T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras](#h2BPTitle)

*   [Introduction](#h3Introduction)
*   [Configuring Your Development Environment](#h3Configuring)
*   [Having Problems Configuring Your Development Environment?](#h3Having)
*   [Project Structure](#h3Project)
*   [Luong’s Attention](#h3Luong)
*   [Encoder Architecture](#h3Encoder)
*   [Decoder Architecture](#h3Decoder)
*   [Input Feeding Approach](#h3Input)
*   [Implementing Luong’s Attention](#h3Implementing)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

The [previous tutorial](https://pyimg.co/kf8ma) on Neural Machine Translation is where we first covered _attention_ (Bahdanau’s attention)_._ Not only did we cover the math behind it, but we also went hands-on training a TensorFlow model from scratch. If you haven’t read it yet, be sure to read that before you go through this tutorial.

![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)

In this tutorial, you will learn how Neural Machine Translation can be achieved using [Luong’s attention](https://arxiv.org/abs/1508.04025) and how it enhances the attention mechanism introduced in [Bahdanau’s attention](https://arxiv.org/abs/1409.0473).

Enough of Google Translate. We also have an interactive demo where you can see the trained model in action.

This lesson is the last in a 2-part series on **NLP 103**:

1.  [_Neural Machine Translation with Bahdanau’s Attention Using TensorFlow and Keras_](https://pyimg.co/kf8ma)
2.  [**_Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras_**](https://pyimg.co/tpf3l) **(this tutorial)**

**To learn how to apply Luong’s attention to the Neural Machine Translation task,** **_just keep reading._**

![Image 2](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20128%20128'%3E%3C/svg%3E)

#### Looking for the source code to this post?

[Jump Right To The Downloads Section](#download-the-code)

* * *

[**Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras**](#TOC)
----------------------------------------------------------------------------------------

* * *

### [**Introduction**](#TOC)

In the academic paper, [Effective Approaches to Attention-Based Neural Machine Translation](https://arxiv.org/abs/1508.04025), Luong et al. provide **more effective** approaches to building attention. We have to note here that the basic intuition behind attention does not change. The additions provided by Luong et al. are _iterative changes_ that make the process of attention **simpler** and **more efficient**.

Building upon the work of [Bahdanau et al.](https://arxiv.org/abs/1409.0473), [Luong et al](https://arxiv.org/abs/1508.04025). add their own twist to the normal attention architecture, suggesting subtle changes to break through the limitations of the old architecture.

**_Note:_** _We will not be diving too deep into the entire code walkthrough since that has already been covered in the Bahdanau’s attention tutorial. This tutorial will cover the significant changes and how to implement them in TensorFlow and Keras._

* * *

### [**Configuring Your Development Environment**](#TOC)

To follow this guide, you need to have the `tensorflow` and `tensorflow-text` installed on your system.

Luckily, TensorFlow is pip-installable:

$ pip install tensorflow==2.8.0
$ pip install tensorflow-text==2.8.0

* * *

### [**Having Problems Configuring Your Development Environment?**](#TOC)

[![Image 3](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20334'%3E%3C/svg%3E)](https://pyimagesearch.com/pyimagesearch-university/)

**Figure 1:** Having trouble configuring your dev environment? Want access to pre-configured Jupyter Notebooks running on Google Colab? Be sure to join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) — you’ll be up and running with this tutorial in a matter of minutes.

All that said, are you:

*   Short on time?
*   Learning on your employer’s administratively locked system?
*   Wanting to skip the hassle of fighting with the command line, package managers, and virtual environments?
*   **Ready to run the code** **_right now_** **on your Windows, macOS, or Linux system?**

Then join [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/) today!

**Gain access to Jupyter Notebooks for this tutorial and other PyImageSearch guides that are** **_pre-configured_** **to run on Google Colab’s ecosystem right in your web browser!** No installation required.

And best of all, these Jupyter Notebooks will run on Windows, macOS, and Linux!

* * *

### [**Project Structure**](#TOC)

We first review our project directory structure.

Start by accessing the **_“Downloads”_** section of this tutorial to retrieve the source code and example images.

From there, take a look at the directory structure:

├── download.sh
├── inference.py
├── output
│   └── loss.png
├── pyimagesearch
│   ├── config.py
│   ├── dataset.py
│   ├── \_\_init\_\_.py
│   ├── loss.py
│   ├── models.py
│   ├── schedule.py
│   └── translator.py
├── requirements.txt
└── train.py

In the `pyimagesearch` directory, we have:

*   `config.py`: The configuration file for the task
*   `dataset.py`: The utilities for the dataset pipeline
*   `loss.py`: Holds the code snippet for the losses needed to train the model
*   `models.py`: Encoder and Decoder for the translation model
*   `schedule.py`: The learning rate scheduler for the training pipeline
*   `translator.py`: The train and inference models

In the core directory, we have four scripts:

*   `download.sh`: A shell script to download the training data
*   `requirements.txt`: The python packages that are required for this tutorial
*   `train.py`: The script run to train the model
*   `inference.py`: The inference script

* * *

### [**Luong’s Attention**](#TOC)

Luong et al. suggest some _small_ but _necessary_ changes to the architecture of the **decoder**. We will first talk about the encoder, the attention layer, and then the decoder. While discussing the architecture, we will also compare it with that of Bahdanau et al. Consider the diagram shown in **Figure 2**.

![Image 4](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20386'%3E%3C/svg%3E)

**Figure 2:** Diagram of the Luong attention architecture (source: Authors).

* * *

### [**Encoder Architecture**](#TOC)

In this paper, the authors opt for a _unidirectional_ (instead of bidirectional as in Bahdanau’s implementation) Recurrent Neural architecture for the encoder. Unidirectional RNNs speed up the computation.

For the encoder we choose a Gated Recurrent Unit (GRU) which takes the present input ![Image 5: x_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2010'%3E%3C/svg%3E) and past hidden state ![Image 6: h_{t-1}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2027%2015'%3E%3C/svg%3E) as input while processing them into the present hidden state ![Image 7: h_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2013%2014'%3E%3C/svg%3E).

![Image 8](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 1**

After passing the entire source sentence to the encoder, we have a set of all the hidden states.

![Image 9](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 2**

* * *

### [**Decoder Architecture**](#TOC)

At each time step, ![Image 10: t](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%206%2010'%3E%3C/svg%3E), in the decoding phase, the main motive is to capture the present hidden state of the decoder, ![Image 11: s_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E), and then to derive a context vector, ![Image 12: c_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%2010'%3E%3C/svg%3E), that captures relevant source-side information.

Specifically, given the target hidden state, ![Image 13: s_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2010'%3E%3C/svg%3E), and the source-side context vector, ![Image 14: c_{t}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2011%2010'%3E%3C/svg%3E), we employ a simple concatenation layer to combine the information from both vectors to produce an attentional hidden state as follows:

![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20288%2067'%3E%3C/svg%3E)

**Equation 3**

The attention vector, ![Image 16: \tilde{s_{t}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2012%2015'%3E%3C/svg%3E), is then fed through the softmax layer to produce the probability of the next decoder word.

![Image 17](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20416%2082'%3E%3C/svg%3E)

**Equation 4**

Enough of mathematical jargon, let’s focus only on the part where the authors have proposed changes to the attention layer.

Bahdanau goes from:

![Image 18](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 5**

Luong goes from:

![Image 19](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%200%200'%3E%3C/svg%3E)

**Equation 6**

* * *

### [**Input Feeding Approach**](#TOC)

With the present proposal, the authors discovered that they were not feeding the attention to the decoder’s recurrent units. This meant that the decoding system did not know which parts of the source sentence were attended to at the previous step.

With that in mind, they now propose to provide the attention to the next decoder unit along with the input and the hidden states. This proved to be a game-changer. While Bahdanau’s model already had this mechanism installed inside of it, Luong’s model had to do it explicitly.

**Figure 3** shows the entire encoding and decoding mechanism envisioned by Luong et al.

[![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20371'%3E%3C/svg%3E)](https://lh6.googleusercontent.com/Akuyf71q90Hyl1mhPDNwIJPvXbxa4XJphM8T7V9AW7xs3nZ7k86mhb2qrEPvzhmSc5HCIYpBJzs5PgGvdvwu-4gXvbgYxhYYL6Q96NP4qzZHlG1o7JdQrNmwvZG7lEgkGVS1rR6XgY4w8BT-u8TuTL0)

**Figure 3:** The animation of neural machine translation with Luong’s attention (source: Authors).

* * *

### [**Implementing Luong’s Attention**](#TOC)

Since all the methods and functions are the same as Bahdanau’s attention, we will discuss Luong’s attention module in this tutorial. The entire code walkthrough is covered in the [previous blog post](https://pyimg.co/kf8ma) of this series.

If you haven’t already, do be sure to read it before proceeding further. And if you have done so already, let us open `models.py` inside the `pyimagesearch` directory and go through it.

class LuongAttention(Layer):
    def \_\_init\_\_(self, attnUnits, \*\*kwargs):
        super().\_\_init\_\_(\*\*kwargs)
        # initialize the attention units
        self.attnUnits = attnUnits
    
    def build(self, inputShape):
        # build the additive attention layer
        self.attention = Attention()

    def get\_config(self):
        # return the configuration of the layer
        return {
            "attnUnits": self.attnUnits,
        }

    def call(self, hiddenStateEnc, hiddenStateDec, mask):
        # grab the source and target mask
        sourceMask = mask\[0\]
        targetMask = mask\[1\]

        # apply attention to align the representations
        (contextVector, attentionWeights) = self.attention(
            inputs=\[hiddenStateDec, hiddenStateEnc, hiddenStateEnc\],
            mask=\[targetMask, sourceMask\],
            return\_attention\_scores=True
        )

        # return the context vector and the attention weights
        return (contextVector, attentionWeights)

We begin by initializing the attention units on **Line 80** inside the `__init__` function of the `LuongAttention` class.

Next, on **Lines 82-84**, we build the attention layer inside the `build` function. We return the attention units inside the `get_config` function **(Lines 86-90)**.

On **Line 92**, we define the `call` method. First, we extract the `sourceMask` and `targetMask` on **Lines 94 and 95**.

On **Lines 98-102**, we apply attention to align the representations. We use the neat attention API from TensorFlow to achieve this. You can read more about it from the official TensorFlow documentation [here](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Attention).

Finally on **Line 105**, we return the `contextVector` and `attentionWeights`.

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

This tutorial walked us through the specific ways Luong’s attention improved the task of Neural Machine Translation. We also learned how to implement the attention module simply using Keras and TensorFlow.

The Bahdanau and Luong attentions are neither state-of-the-art techniques nor widely used in NLP systems. But they do provide a stepping stone and a concrete foundation for what comes next.

We will be back with another tutorial on attention and its uses, but till that time, here’s a little food for thought:

“What happens if we only use attention in an architecture? ?”

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “Neural Machine Translation with Luong’s Attention Using TensorFlow and Keras,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, [https://pyimg.co/tpf3l](https://pyimg.co/tpf3l)

@incollection{ARG-RR\_2022\_Luong,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {Neural Machine Translation with {Luong’s} Attention Using {TensorFlow} and {Keras}},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/tpf3l},
}

* * *

![Image 21: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

**To download the source code to this post (and be notified when future tutorials are published here on PyImageSearch), _simply enter your email address in the form below!_**

![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Download the Source Code and FREE 17-page Resource Guide

Enter your email address below to get a .zip of the code and a **FREE 17-page Resource Guide on Computer Vision, OpenCV, and Deep Learning.** Inside you'll find my hand-picked tutorials, books, courses, and libraries to help you master CV and DL!