Title: A Deep Dive into Transformers with TensorFlow and Keras: Part 2 - PyImageSearch

URL Source: https://pyimagesearch.com/2022/09/26/a-deep-dive-into-transformers-with-tensorflow-and-keras-part-2/

Published Time: 2022-09-26T13:00:00+00:00

Markdown Content:
* * *

**Table of Contents**
---------------------

*   [A Deep Dive into Transformers with TensorFlow and Keras: Part 2](#h2BPTitle)

*   [A Brief Recap](#h3Recap)
*   [The Land of Attention](#h3Attention)
*   [Connecting Wires](#h3Wires)
*   [Skip Connections](#h3Connections)
*   [Layer Normalization](#h3Normalization)
*   [Feed-Forward Network](#h3FFN)
*   [Positional Encoding](#h3Positional)

*   [Summary](#h2Summary)

*   [Citation Information](#h3Citation)

* * *

In this tutorial, you will learn about the connecting parts of the Transformers architecture that hold together the encoder and decoder.

[![Image 1](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%201080%20607'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/Transformers-Part-2-Featured.png)

This lesson is the 2nd in a 3-part series on **NLP 104**:

1.  [_A Deep Dive into Transformers with TensorFlow and Keras: Part 1_](https://pyimg.co/8kdj1)
2.  [**_A Deep Dive into Transformers with TensorFlow and Keras: Part 2_**](https://pyimg.co/pzu1j) **(today’s tutorial)**
3.  _A Deep Dive into Transformers with TensorFlow and Keras: Part 3_

**To learn how the transformers architecture stitches together the multi-head attention layer with other components,** **_just keep reading._**

* * *

[**A Deep Dive into Transformers with TensorFlow and Keras: Part 2**](#TOC)
---------------------------------------------------------------------------

* * *

### [](#TOC)[**A Brief Recap**](#TOC)

In the [previous tutorial](https://pyimg.co/8kdj1), we studied the evolution of attention from its simplest form into Multi-Head Attention as we see today. In **Video 1**, we illustrate how the **Input** matrix is projected into the **Query**, **Key**, and **Value** Matrices.

![Image 2: Video Thumbnail](https://embed-ssl.wistia.com/deliveries/e6ade5b75a9d2c4edb90c80cc2144ac45da3b845.webp?image_crop_resized=960x540)

**Video 1:** Projecting the Input matrix into the Query, Key, and Value matrices.

Here, the Input matrix is represented through the initial gray matrix ![Image 3: I](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2011'%3E%3C/svg%3E). The matrix placed below matrix ![Image 4: I](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%208%2011'%3E%3C/svg%3E) is our weight matrix (i.e., ![Image 5: W_Q](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2016'%3E%3C/svg%3E) (red), ![Image 6: W_K](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2026%2014'%3E%3C/svg%3E) (green), and ![Image 7: W_V](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2024%2014'%3E%3C/svg%3E) (blue), respectively). As the Input is multiplied by these three weight matrices, the Input is projected to produce the Query, Key, and Value matrices, shown with red, green, and blue colors, respectively.

* * *

### [**The Land of Attention**](#TOC)

Our three friends, **Query**, **Key**, and **Value**, are the central actors that bring Transformers to life. In **Video 2**, we build an animation that shows how the attention score is _computed_ from the Query, Key, and Value matrices.

![Image 8: Video Thumbnail](https://embed-ssl.wistia.com/deliveries/8a245d9c28e10c8ceb9c8cf189334856f1ee5e19.webp?image_crop_resized=960x540)

****Video 2:**** Animation of the Attention module.

First, the Query and Key matrices are multiplied to arrive at a product term. Let us call this term the ![Image 9: QK](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2028%2014'%3E%3C/svg%3E) Product. Next, we scale the product term with ![Image 10: 1/\sqrt{\text{dim}}](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2057%2019'%3E%3C/svg%3E); this was done to prevent the vanishing gradient problem, as explained in our [previous blog post](https://pyimagesearch.com/2022/09/05/a-deep-dive-into-transformers-with-tensorflow-and-keras-part-1/#h5Solution). Finally, the scaled ![Image 11: QK](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2028%2014'%3E%3C/svg%3E) Product term is passed through a softmax layer. The resultant output is multiplied by the value matrix to arrive at the final attention layer.

The entire animation is neatly encapsulated in the attention equation, as shown in **Equation 1**.

[![Image 12](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20152'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/attention-eqn.png)

**Equation 1**

Let’s now talk about a _problem_ with the above module. As we have [seen earlier](https://pyimagesearch.com/2022/09/05/a-deep-dive-into-transformers-with-tensorflow-and-keras-part-1/#h4V5), the attention module can be easily extended to Self-Attention. In a Self-Attention block, the Query, Key, and Value matrices come from the **same source**.

Intuitively the attention block will attend to _each_ token of the inputs. Take a moment and think about what can possibly go wrong here.

In NMT (Neural Machine Translation), we predict a target token in the decoder given a set of previously decoded target tokens and the input tokens.

If the decoder already has access to all the target tokens (both previous and future), it will not **learn to decode**. We would need to mask out the target tokens that are not yet decoded by the decoder. This process is necessary to have an **auto-regressive** decoder.

[Vaswani et al.](https://arxiv.org/abs/1706.03762) made a _minute_ modification to this attention pipeline to accomplish masking. Before passing the scaled ![Image 13: QK](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2028%2014'%3E%3C/svg%3E) product through a softmax layer, they would mask certain parts of the product with a large number (e.g., negative infinity).

The change is visualized in **Video 3** and termed Masked Multi-Head Attention or MMHA.

![Image 14: Video Thumbnail](https://embed-ssl.wistia.com/deliveries/0887f910ed47154a25d485a589f57b441808ca0c.webp?image_crop_resized=960x540)

**Video 3:** Animation of the Masked-Multi-Head-Attention module.

* * *

### [**Connecting Wires**](#TOC)

At this point, we have covered the most **important** building block of the Transformer architecture, **attention**. But despite what the paper title says ([Attention Is All You Need](https://arxiv.org/abs/1706.03762)), only attention cannot create the entire model. We need connecting wires to hold each piece together, as shown in **Figure 1**.

[![Image 15](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20723%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/entire-architecture-1.png)

**Figure 1:** A diagram of the entire Transformer architecture.

The connecting wires that hold the architecture together are:

*   Skip Connection
*   Layer Normalization
*   Feed-Forward Network
*   Positional Encoding

* * *

### [**Skip Connections**](#TOC)

Intuitively skip connections introduce the representation of a previous stage into a later stage. This allows the raw representation prior to a sub-layer to be injected along with the output from the sub-layer. The skip connections in the Transformers architecture are highlighted with red arrows in **Figure 2**.

[![Image 16](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20723%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/skip-architecture.png)

**Figure 2:** A diagram showing only the skip connections in the architecture.

Now, this raises the question, **why is it important?**

Did you notice that in this blog post series, every time a part of the architecture is mentioned, we bring up the original figure of the entire architecture shown at the very **beginning**? This is because we process the information better when we reference it with information received at an **earlier** stage.

Well, it turns out the Transformer works the same way. Creating a mechanism to add representations of past stages into later stages of the architecture allows the model to process the information better.

* * *

### [**Layer Normalization**](#TOC)

The official TensorFlow documentation of [Layer Normalization](https://www.tensorflow.org/api_docs/python/tf/keras/layers/LayerNormalization) says:

> Notice that with Layer Normalization, the normalization happens across the axes within each example rather than across different examples in the batch.

Here, the input is a matrix ![Image 17: X](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2014%2011'%3E%3C/svg%3E) with ![Image 18: n](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) rows where ![Image 19: n](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%2010%207'%3E%3C/svg%3E) is the number of words in the input sentence. The normalization layer is a row-wise operation that normalizes each row with the same weights.

In NLP tasks, as well as in the transformer architecture, the requirement is to be able to calculate the statistics across each feature dimension and instance independently. Thus Layer Normalization makes more intuitive sense than Batch Normalization. The highlighted parts in **Figure 3** show the Layer Normalization.

[![Image 20](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20723%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/normalization-architecture.png)

**Figure 3:** A diagram showing only the normalization layers in the architecture.

We would also like to bring forward a [stack exchange discussion](https://stats.stackexchange.com/a/505349) thread about why Layer Normalization works in the Transformer architecture.

* * *

### [**Feed-Forward Network**](#TOC)

Each encoder and decoder layer consists of a fully connected feed-forward network. The purpose of this layer is straightforward, as shown in **Equation 2**.

[![Image 21](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%2069'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/ffn.png)

**Equation 2**

The idea is to project the attention layer output into a higher dimensional space. This essentially means the representations are stretched to a higher dimension, so their details are magnified for the next layer. The application of these layers in the Transformers architecture is shown in **Figure 4**.

[![Image 22](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20723%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/ffn-architecture.png)

**Figure 4:** A diagram showing only the feed-forward network (FFN) in the architecture.

Let us pause here and take a look back at the architecture.

*   We studied the encoder and the decoder
*   The evolution of attention, as we see in [Vaswani et al.](https://arxiv.org/abs/1706.03762)
*   Skip connections for contextualization
*   Layer normalization
*   And finally, feed-forward networks

But there is still a part of the entire architecture that is missing. Can we guess what it is from **Figure 5**?

[![Image 23](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20723%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/positional-encoding-architecture.png)

**Figure 5:** Can you identify the missing piece of the puzzle?

* * *

### [**Positional Encoding**](#TOC)

Now, before we understand positional encoding and why it has been introduced in the architecture, let us first look at what our architecture achieves in contrast to sequential models like RNN and LSTM.

The transformer can attend to parts of the input tokens. The encoder and decoder units are built out of these attention blocks, along with non-linear layers, layer normalization, and skip connections.

But RNNs and other sequential models had something that the architecture still lacks. It is to understand the **order** of the data. So how do we introduce the order of the data?

1.  We can add attention to a sequential model, but then it will be similar to [Bahdanau’s](https://pyimg.co/kf8ma) or [Luong’s](https://pyimg.co/tpf3l) attention which we have already studied. Furthermore, adding a sequential model will beat the purpose of using _only attention units_ for the task at hand.
2.  We can also inject the data order into the model along with the input and target tokens.

[Vaswani et al.](https://arxiv.org/abs/1706.03762) opted for the 2nd option. This meant encoding the position of each token and injecting it with the input. So, now there are a couple of ways we can achieve positional encoding.

The first is to simply encode everything in a **binary encoding scheme**, as shown in **Figure 6**. Consider the following code snippet where we visualize the binary encoding of a range of numbers.

*   `n` corresponds to the range of `0-n` positions
*   `dims` corresponds to the dimension to which each position is encoded

[![Image 24](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20497%20500'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/binary-snippet.png)

**Figure 6:** Code Snippet showing binary encoding.

The output of the code snippet in **Figure 6** is shown in **Figure 7**.

[![Image 25](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20400%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/binary-pos.png)

******Figure 7:****** Binary encoding output from the code snippet.

[Vaswani et al.](https://arxiv.org/abs/1706.03762), however, introduced a sine and cosine function that encodes the sequence position according to **Equations 3 and 4**:

[![Image 26](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20253'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/sin.png)

**Equation 3**

[![Image 27](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20600%20235'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/cos.png)

**Equation 4**

Here, `pos` refers to the position, and `i` is the dimension. **Figure 8** is the code to implement the equations.

[![Image 28](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20796%201024'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/sinusoid-snippet.png)

**Figure 8:** Code snippet showing sinusoidal encoding.

The output of the code in **Figure 8** is shown in **Figure 9**.

[![Image 29](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20400%20400'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/sin-pos.png)

**Figure 9:** Output of sinusoidal encoding from the code snippet.

The two encodings look similar, don’t they? Positional encoding is a system in which each position is encoded into a vector. An important thing to note here is that each vector should be unique.

Binary encoding seems to be a perfect candidate for positional encoding. The problem with the binary system is that of `0`s. This means binary is discrete in nature. Deep Learning models, on the other hand, love to work with continuous data.

The reason for using sinusoidal encoding is thus three-fold.

[![Image 30](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20700%20393'%3E%3C/svg%3E)](https://pyimagesearch.com/wp-content/uploads/2022/09/positional-encoder.gif)

**Figure 10:** Positional Encoder.

* * *

### What's next? We recommend [PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend).

![Image 31: Video Thumbnail](https://embed-ssl.wistia.com/deliveries/4bda0a1602c8b4d96d63a02617f3069e.webp?image_crop_resized=640x360)

**Course information:**  
84 total classes • 114+ hours of on-demand code walkthrough videos • Last updated: February 2024  
★★★★★ 4.84 (128 Ratings) • 16,000+ Students Enrolled

**I strongly believe that if you had the right teacher you could _master_ computer vision and deep learning.**

Do you think learning computer vision and deep learning has to be time-consuming, overwhelming, and complicated? Or has to involve complex mathematics and equations? Or requires a degree in computer science?

That’s _not_ the case.

All you need to master computer vision and deep learning is for someone to explain things to you in _simple, intuitive_ terms. _And that’s exactly what I do_. My mission is to change education and how complex Artificial Intelligence topics are taught.

If you're serious about learning computer vision, your next stop should be PyImageSearch University, the most comprehensive computer vision, deep learning, and OpenCV course online today. Here you’ll learn how to _successfully_ and _confidently_ apply computer vision to your work, research, and projects. Join me in computer vision mastery.

**Inside PyImageSearch University you'll find:**

*   ✓ **84 courses** on essential computer vision, deep learning, and OpenCV topics
*   ✓ **84 Certificates** of Completion
*   ✓ **114+ hours** of on-demand video
*   ✓ **Brand new courses released _regularly_**, ensuring you can keep up with state-of-the-art techniques
*   ✓ **Pre-configured Jupyter Notebooks in Google Colab**
*   ✓ Run all code examples in your web browser — works on Windows, macOS, and Linux (no dev environment configuration required!)
*   ✓ Access to **centralized code repos for _all_ 536+ tutorials** on PyImageSearch
*   ✓ **Easy one-click downloads** for code, datasets, pre-trained models, etc.
*   ✓ **Access** on mobile, laptop, desktop, etc.

[Click here to join PyImageSearch University](https://pyimagesearch.com/pyimagesearch-university/?utm_source=blogPost&utm_medium=bottomBanner&utm_campaign=What%27s%20next%3F%20I%20recommend)

* * *

[**Summary**](#TOC)
-------------------

Now at this point, almost everything about transformers is known to us. But it is also important to pause and ponder the necessity of such an architecture.

In 2017, the answer lay in their paper. [Vaswani et al.](https://arxiv.org/abs/1706.03762) mention their reasons for choosing the simpler version of other transduction models published at the time. But we have come a long way since 2017.

In the last 5 years, transformers have absolutely taken over Deep Learning. There is no task, nook, or corner safe from attention applications. So what made this model so good? And how was that conceptualized in a simple Machine Translation task?

Firstly, Deep Learning researchers are not soothsayers, so there was no way of knowing that Transformers would be so good at every task. But the architecture is intuitively simple. It works by attending to parts of the input tokens that are relevant to parts of the target token. This approach worked wonders for Machine Translation. But will it not work on other tasks? Image Classification, Video Classification, Text-to-Image generation, segmentation, or volume rendering?

Anything that can be expressed as tokenized input being mapped to a tokenized output set falls in the realm of Transformers.

This brings the theory of Transformers to an end. In the next part of this series, we will focus on creating this architecture in TensorFlow and Keras.

* * *

### [**Citation Information**](#TOC)

**A. R. Gosthipaty and R. Raha.** “A Deep Dive into Transformers with TensorFlow and Keras: Part 2,” _PyImageSearch_, P. Chugh, S. Huot, K. Kidriavsteva, and A. Thanki, eds., 2022, [https://pyimg.co/pzu1j](https://pyimg.co/pzu1j)

@incollection{ARG-RR\_2022\_DDTFK2,
  author = {Aritra Roy Gosthipaty and Ritwik Raha},
  title = {A Deep Dive into Transformers with {TensorFlow} and {K}eras: Part 2},
  booktitle = {PyImageSearch},
  editor = {Puneet Chugh and Susan Huot and Kseniia Kidriavsteva and Abhishek Thanki},
  year = {2022},
  note = {https://pyimg.co/pzu1j},
}

* * *

![Image 32: Featured Image](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20500%20276'%3E%3C/svg%3E)

### Unleash the potential of computer vision with Roboflow - Free!

*   Step into the realm of the future by [signing up or logging into your Roboflow account](https://roboflow.com/?ref=pyimagesearch). Unlock a wealth of innovative dataset libraries and revolutionize your computer vision operations.
*   Jumpstart your journey by choosing from our broad array of datasets, or benefit from PyimageSearch’s comprehensive library, crafted to cater to a wide range of requirements.
*   Transfer your data to Roboflow in any of the 40+ compatible formats. Leverage cutting-edge model architectures for training, and deploy seamlessly across diverse platforms, including API, NVIDIA, browser, iOS, and beyond. Integrate our platform effortlessly with your applications or your favorite third-party tools.
*   Equip yourself with the ability to train a potent computer vision model in a mere afternoon. With a few images, you can import data from any source via API, annotate images using our superior cloud-hosted tool, kickstart model training with a single click, and deploy the model via a hosted API endpoint. Tailor your process by opting for a code-centric approach, leveraging our intuitive, cloud-based UI, or combining both to fit your unique needs.
*   Embark on your journey today with absolutely no credit card required. Step into the future with Roboflow.

[Join Roboflow Now](https://roboflow.com/?ref=pyimagesearch)

* * *

![Image 33](data:image/svg+xml,%3Csvg%20xmlns='http://www.w3.org/2000/svg'%20viewBox='0%200%20410%20567'%3E%3C/svg%3E)

#### Join the PyImageSearch Newsletter and Grab My FREE 17-page Resource Guide PDF

Enter your email address below to **join the PyImageSearch Newsletter** and **download my FREE 17-page Resource Guide PDF** on Computer Vision, OpenCV, and Deep Learning.