Intuition Behind Transformers Architecture in NLP.
By: Oleg Borisov
Tags: Machine Learning, Getting Started, NLP, Editors Pick, Transformers

Time Series Analogy.

As we all know, language has a sequential nature, and order of the words matters. Similarly to the language, time series also have this importance of the order in which data points occur. So, let’s take a look at one time-series analysis example and check if we could apply similar ideas to the NLP task. Say we have some time-series data as shown below:

Time series with noise. Image by the Author

In this example, we have some stock price and its value changes over the time, we would like to analyse the trend, unfortunately, our data is noisy, and hence it is a bit hard to extract the trend.

In statistics, it is known that we could apply convolution operation using some Kernel in order to reduce the noise. It could be done as shown below for some point x_i.

Green are the time-series data points, Yellow is the Kernel which we use to convolve. Image taken from http://www.invasivecode.com/weblog/convolutional-neural-networks-ios-10-macos-sierra/

After the convolution we will get a denoised representation y_i of our initial data point x_i. If we apply convolution to our example graph, using the Kernel [0.25, 0.5, 0.25] we will get.

Denosied time-series. Image by the Author

As you can clearly see, the overall trend is that the stock price is raising.

Ok, sounds great, shall we apply some similar idea in NLP domain? In fact, yes, we can do it, and Convolutional Neural Networks have been using this approach while learning the values for the kernel matrix, but there is a huge drawback!

In the time-series example we have considered the neighbouring points are correlated with each-other thus we could smoothen the graph, unfortunately , the same thing does not apply to the language (at least not always). Consider example below:

When Ben and I were playing football, John called me and asked to help him on the weekend to assemble his new furniture.

In this case, if we do not use a large window size, there is no possibility for the system to understand to whom the words: “him/his”, are referring to. In some languages like german, it could happen that you would have a very large sentence, and the verb is going to be in the end, so sometimes you would only guess what is happening in the sentence until you read the verb in the end.

Therefore, while, the idea of moving to the representation space is very good, using convolution/window/kernel approach to do that is not applicable for the NLP tasks. So let’s think if there exists some other approach that we could use, and let’s focus on idea of so called self-attention.

Self-Attention.

In self-attention, the main goal is to understand, how the words in a sentence are interconnected with each other. In the example sentence we considered earlier, if we were to ask question “Doing what?” then word Ben should be connected (using terminology attend) to the phrase playing football. similarly words him/his are connected to the word John, as he was the one to ask for help and his furniture needs to be assembled. As we have seen window approach is not applicable here, so let us take a look at some other approach.

Thinking in terms of NLP Embeddings.

In one of my previous articles, I have talked about a very cool and useful way of representing the words called Embeddings. One of the nice properties of word embeddings is that if two words are similar than their cosine similarity is going to be equal to 1.

For simplicity, consider that we have a 4 word sentence (hence we work with 4 tokens). Let us, first convert our tokens to the vector representation using some embedding so that we get tokens x_i.

Image by the Author

Now, for the sake of simplicity of an example, let’s convert vector x_2 to the representation space y_2, using all our word-vectors x_1, x_2, x_3, x_4.

Similarly as in the time-series example we are going to do as shown below:

Where we know what the x_i values are, and need to figure out how to compute weights w_{ij} as we do not want to use any hardcoded values and want to make sure that weights can reflect similarities of words given the context of the sentence. One of the possible ways to do that could be dot product between two vectors (if two vectors are independent dot product would be zero, if they are similar dot product will be a large positive number). So, let’s do it this way and define weight w_{ij} as:

And, since in this example we are working with vector x_2, I will update our schematic diagram as shown below

Image by the Author

Now, some weights are going turn out to be super large, some of them will turn out to be relatively small, so to make our model consistent, we can normalise the weights to make sure that they sum to 1.

Image by the Author

And now, after the normalisation, exactly these weights w_{i2} are going to be used to compute representation vector y_2.

And thus the final diagram would look like:

Image by the Author

Now we have found y_2, by querying x_2 and we can apply similar procedure to find representation vectors for other tokens: y_1, y_3, y_4. And to align out graph with Transformers terminology in this case: Query, Key Values are marked as in figure below.

Image by the Author

As you could have noticed, this is a very robust and easy to perform method, which did not even include any Neural Network and learning parameters at all (Even though I used word weights to make analogy with time-series example, they are not connected with NN weights).

But… aren’t we a little bit naive by just using this simple architecture to reflect self-attention? Probably yes, as in this case it is still might be complicated for us to say wether word “him” refers to Bob or to John. Therefore, in this architecture, we only rely on the quality of our Embedding, and if it is bad, then so are our results going to be.

So, what could be done in this case? How about we would introduce some simple Feedforward Neural Network at the points of our Key, Query and Value parts. Thus our original architecture could be slightly updated as follows:

Self-Attention Architecture. Figure by Oleg Borisov

And this is what actually happens behind the Transformer’s so called “Scaled Dot-Product Attention”. Which in paper is shown like below.

Figure taken from original paper “Attention Is All You Need” written by A. Vaswani et al.

Where above, Q is matrix that contains queries, K — keys, and V — values. Which we get by passing words embeddings from our sentence through the respective Feedforward NN.

The only things that were not discussed so far are: the scaling part and the mask. The scaling part is needed just to stabilise the gradients. Mask is used in the decoding part during the training process, to prevent us from attending the words that the is not supposed to see yet (If you still do not understand it don’t worry, for now it is not that important).

And what is the beauty of it, is that the whole computation could be completely done in a parallel manner, as we can supply the entire sentence to the model which is going to perform all the computation simultaneously!

And in fact this complicated diagram can be easily simplified and shown as formula:

There is another question that one could ask: is having one attention enough for us? Unfortunately, no, that is why we need multiple attention mechanisms.

Multi-Head Attention.

Consider the following sentence:

Today my friend John bought a car.

From this sentence, let’s consider what words could be related/connected to the word “bought”:

Bought what? --> car

Who bought? --> John

When? --> today

Thickness of arrow depicts the strength/importance of the word to the query word. Image by the Author

As you can see we in the language we usually have different words related to the query word depending on the “question” that we might be asking. Having only one self-attention is going to oversaturate our mechanism and it might become too complicated for us to find what word matters the most.

To make analogy with Convolutional Neural Networks, we have multiple kernels in each layer, where each of the kernels learns their own unique features.

So, instead of using one self-attention, let’s use multiple of those, and this combined block is called called Multi-Headed Attention.

Let’s now take a moment and see how that works, as it also has some important details that one needs to understand.

Figure modified from original paper “Attention Is All You Need” written by A. Vaswani et al.

As you can see from the image above, we have sentence X (which is a matrix that contains stacked word embeddings x_1, x_2, …). Which we pass through the Linear layer and get our Key, Query and Value matrices, which we then supply to the Scaled Dot-Product Attention as described. The only difference now, is that we need to perform all those operations h times, where h corresponds to the number of heads that we define.

Now, there is one important part that I would like to focus our attention on.

Each Head does not process the whole embedding vector, it processes just a part of the vector.

Let me illustrate:

Image by the Author.

Assume that our embedding is of size d, and that we have h heads, that means that the first head is going to process the first d/h dimensions of the vector, the second head is going to process the next d/h dimensions, and we continue in the same pattern

Why is this done? As we know from the embeddings article each dimension of the embedding space could represent some special information (if a word represented has wings or is a type of bird) or some abstract information information like:

what part of speech the word belongs to

what hyperplane between two axis represents, etc.

So it is possible also that the embedding could also learn to reflect different type of information that is going to help us to distinguish the connection between different words in a sentence. And using this approach, for example one of the Heads will be able to answer the question “who?”, the other Head will answer question “did what?”, some other Head will learn connection to the question “when?” etc.

After the split of embeddings is done and all the words are passed through the Scaled Dot-Product Attention layer, we will concatenate the output vectors from all our heads, to make sure that for each word in our sentence we can answer a particular question (that head has managed to learn).

In the end we make another pass through the Linear FF layer to make sure that our output has some particular dimension.

This is what the Multi-Head attention is about, which is actually the main, the hardest and the most confusing part of transformer, which turned out to be not too complicated!

Encoder architecture.

Now, let’s look at the global picture, focus on the encoder architecture of the transformer (In this article I am not going to touch on decoder, as it has a similar structure, with some little differences which could be omitted for this article).

Encoder architecture. Figure taken from original paper “Attention Is All You Need” written by A. Vaswani et al.

As the first step, our original text is passed through the Input Embedding which is straight forward word-vector representation part. Then we need to sum it up with the Positional Encoding, which is crucial for transformer as we process all tokens all together at the same time, so this allows the transformer to know the order of the tokens inside of the sentence

We can either train it from scratch or use a deterministic approach by using some special formula

Formula used in “Attention is all you need” paper

After that, we just our vectors to the Transformer block which on the figure shown as the grey box. This Transformer block is repeated N times to allow us to have deeper architecture, so output of block 1 is used as input for the block 2.

As you can see when the input enters the transformer block, it passes through the Multi-Headed Attention mechanism (which we now understand very well), but you notice that there is a residual connection, that also passes our input vector around the Multi-Headed Attention block. Why is it there?

Residual connections.

Think about the training process for a second. Imagine that we are now in a process of updating the weights of the NN and a gradient signal comes in

Gradient signal and updating the weights during the training process. Figure modified from original paper “Attention Is All You Need” written by A. Vaswani et al.

The main problem here is that since we have N transformer blocks, h Multi-Headed Attention blocks in each transformer blocks, then we risk having a vanishing gradient problem, and which might result in a problem of not training our NN at all.

So we can consider those residual connections to be very important optimisation approach as well as the trick that helps us to avoid vanishing gradients problem.

Final remarks.

So as we have seen in this article is that Transformer architecture is a very neat and beautiful solution that allowed to make some incredible progress in NLP. I really like to think about Transformers as of the undirectional architecture as we can directly supply to it the whole sentence, and the NN will be able to use its Self-Attention mechanism to find the dependencies between the words in a sentence no matter how far apart those words are located in the sentence.

LSTMs and Bi-LSTMs might have hard time to catch on those word relationships in the sentence, because of the sequential nature of training them.

Transformers are much easier and much faster to train as the batch-wise operations are very natural using this architecture. While with RNNs to do some similar operations would require one to implement some special tricks which might be quite complex and sometimes counterintuitive.

I hope after reading this article you have managed to understand that Transformer is a quite simple architecture once you understand the Self-Attention part, and that there is nothing to be afraid of!

Stay tuned for more NLP and Machine Learning articles, stay safe and enjoy your holidays!