Multi_Head Attention in BERT different from Transformer?
Author: PedroUria
Pull request: no

I have been digging through the code to understand the whole architecture of BERT (great job by the way, it's really easy to follow), and I noticed the way Multi-Headed Attention is implemented is different than from the original Transformer (unless I'm missing something). In particular, instead of using learnable weights to project the original keys, queries and values into different subspaces, they are just broken up into smaller vectors, each with different components of the originals. I am referring to the `self.transpose_for_scores` method of the `BertSelfAttention` class.

I was just wondering if there is any reason for this, as I have not seen it mentioned on the original paper. Maybe there would just be too many parameters if they included those weights?

Comments:
Hi!

In the forward pass of the BertSelfAttention model you’re getting the hidden state of the previous layer which is of size `(batch_size, sequence_length, 768)` (768 being the embedding dimension).

The first step of the attention is to obtain the `mixed_query_layer`, `mixed_key_layer` as well as the `mixed_value_layer`, which are all of size `(batch_size, sequence_length, 768)`.

The 768 here isn’t actually directly related to the embedding size, but it is related to the number of heads (12) and the dimension of the query/key/value (64) vectors (12 * 64 = 768).

What we’re doing in the `transpose_for_scores` function is that we are reshaping our query/key/value layers so that they are of shape `(batch_size, number_of_heads, sequence_length, qkv_dimension)` -> `(batch_size, 12, sequence_length, 64)`.

It is then easy to compute the attention scores and apply the attention mask.

Is that helpful?

Thanks for the answer! I think I understand the code, but if you take a look at the [equation](https://imgur.com/a/WdqVG3J) from the [transformer paper](https://arxiv.org/pdf/1706.03762.pdf), here Q = `mixed_query_layer `, K = `mixed_key_layer ` and V = `mixed_value_layer`, and each of them are being multiplied by a different weight W_i^Q, W_i^K and W_i^V for each attention head i. 

I don't see any equivalent to these weights on your code, instead as you say you just reshape Q, K and V, do the self-attention on each Q_i, K_i, V_i and then concat and multiply by W^0 (`BertSelfOutput`).  I have yet to look at the transformer code, so maybe the notation in the paper is misleading and they actually did exactly what BERT is doing?

I believe that our implementation respects the formula in the paper. It is indeed Google's own implementation for BERT, you can check out their code and how they computed the attention scores here:

Resizing the [query layer](https://github.com/google-research/bert/blob/master/modeling.py#L690-L692)
Resizing the [key layer](https://github.com/google-research/bert/blob/master/modeling.py#L695-L696)
Resizing the [values layer](https://github.com/google-research/bert/blob/master/modeling.py#L727-L729)

Our BERT code is very similar to the original TF code to make the import/export of weights easy, so you would find the same ideas in both implementations.

Yes, I checked the code of the Transformer and you are right, the Multi-Head Attention is implemented in the exact same way as BERT (both original and this repo, of course). The Transformer paper explains a slightly different Multi-Head Attention, at least to my understanding, and it actually looks more powerful. Anyway, closing this issue as my doubt has been solved. Thanks again for your answers!

Glad I could help!

I had the same question, so I followed both steps in the implementation and paper. Since BertSelfAttention computes all heads in parallel, they look equivalent.

![image](https://user-images.githubusercontent.com/51022522/192971602-1f336524-5ae6-4ca3-ae7d-5fc2e06843f6.png)


Hi, I may have overlooked something, but while reading through the code for the attention mechanism in BERT, I couldn't find the multiplication of a `W_output` with the concatenation of the heads' output, as described in the original paper. Have you merged this weight with the `W_value`? or with the next FF layer (since these are two sequential linear multiplication)?

That multiplication is implemented here.

https://github.com/huggingface/transformers/blob/960807f62e53676723ab8281019219864ef3db4d/src/transformers/models/bert/modeling_bert.py#L386