1 | Basics of Self-Attention
Tags: Transformers, one concept at a time, Machine Learning, Data Science, Tocaat, NLP

Contents:

1. Introduction

Transformers are an ML architecture that have been used successfully in a wide variety of NLP tasks, especially sequence to sequence (seq2seq) ones such as machine translation and text generation. In seq2seq tasks, the goal is to take a set of inputs (e.g. words in English) and produce a desirable set of outputs (- the same words in German). Since their inception in 2017, they’ve usurped the dominant architecture of their day (LSTMs) for seq2seq and have become almost ubiquitous in any news about NLP breakthroughs (for instance OpenAI’s GPT-2 even appeared in mainstream media!).

Fig 1.1 — machine translation (EN → DE)⁴

This column is intended as a very gentle, gradual introduction to the math, code and concept behind Transformer architecture. There’s no better place to start with than the attention mechanism because:

The most basic transformers rely purely on attention mechanisms³.

2. Self-Attention — the math

We want an ML system to learn the important relationships between words, similar to the way a human being understands words in a sentence. In Fig 2.1 you and I both know that “The” is referring to “animal” and thus should have a strong connection with that word. As the diagram’s colour coding shows, this system knows that there is some connection between “animal”, “cross”,“street” and “the” because they’re all related to “animal”, the subject of the sentence. This is achieved through Self-Attention.⁴

Fig 2.1 — which words does “The” pay attention to?⁴

At its most basic level, Self-Attention is a process by which one sequence of vectors x is encoded into another sequence of vectors z (Fig 2.2). Each of the original vectors is just a block of numbers that represents a word. Its corresponding z vector represents both the original word and its relationship with the other words around it.

Fig 2.2: sequence of input vectors x getting turned into another equally long sequence of vectors z

Vectors represent some sort of thing in a space, like the flow of water particles in an ocean or the effect of gravity at any point around the Earth. You can think of words as vectors in the total space of words. The direction of each word-vector means something. Similarities and differences between the vectors correspond to similarities and differences between the words themselves (I’ve written about the subject before here).

Let’s just start by looking at the first three vectors and only looking in particular at how the vector x2, our vector for “cat”, gets turned into z2. All of these steps will be repeated for each of the input vectors.

First, we multiply the vector in our spotlight, x2, with all the vectors in a sequence, including itself. We’re going to do a product of each vector and the transpose (the diagonally flipped version) of x2 (Fig 2.3). This is the same as doing a dot product and you can think of a dot product of two vectors as a measure of how similar they are.

Fig 2.3: transposed multiplication (superscript “T” = “transposed”)

The dot product of two vectors is proportional to the cosine of the angle between them (Fig 2.4) — so the more closely they align in direction, the larger the dot product. If they were pointing in the exact same direction then the angle A would be 0⁰ and a cosine of 0⁰ is equal to 1. If they were pointing in opposite directions (so that A = 180⁰) then the cosine would be -1.

Fig 2.4 — dot product of two vectors

As an aside, note that the operation we use to get this product between vectors is a hyperparameter we can choose. The dot product is just the simplest option we have and the one that’s used in Attention Is All You Need³ (AIAYN).

If you want an additional intuitive perspective on this, Bloem’s¹ post discusses how self-attention is analogous to the way recommender systems determine the similarity of movies or users.

So we put one word under the spotlight at a time and determine its output from its neighbourhood of words. Here we’re only looking at the words before and after but we could choose to widen that window in the future.

Fig 2.5 — raw weights for each j-th vector

If the spotlit word is “cat”, the sequence of words we’re going over is “the”, “cat”, “sat”. We’re asking how much attention the word “cat” should pay to “the”, “cat” and “sat” respectively (similar to what we see in Fig 2.1).

Multiplying the transpose of our spotlit word vector and the sequence of words around it will give us a set of 3 raw weights (Fig 2.5). Each weight is proportional to how connected the two words are in meaning. We need to then normalise them so they are easier to use going ahead. We’ll do this using the softmax formula (Fig 2.6). This converts a sequence of numbers to be within the range of 0, 1 where each output is proportional to the exponential of the input number. This makes our weights much easier to use and interpret.

Fig 2.6: normalising raw weights via softmax function

Now we take our normalised weights (one per every vector in the j sequence), multiply them respectively with the x input vectors, sum the products and bingo! We have an output z vector, (Fig 2.5)! This is, of course, the output vector just for x2 (“cat”) — this operation will be repeated for every input vector in x until we get the output sequence we saw in Fig 2.2.

Fig 2.7: Final operation to get our new sequence of vectors, z

This explanation so far may have raised some questions:

Aren’t the weights we calculated highly dependent on how we determined the original input vectors?

Why are we relying on the similarity of the vectors? What if we want to find a connection between two ‘dissimilar’ words, such as the object and subject of “The cat sat on the matt”?

In the next post, we’ll address these questions. We’ll transform each vector for each of its different uses and thus define relationships between words more precisely so that we can get an output more like Fig 2.8.