Initializing attention weights in T5
Author: Arij-Aladel
Pull request: no

@patrickvonplaten @patil-suraj @craffel 
Excuse me if this question is repeated but I did not find an answer for it 

In these lines 

```
        elif isinstance(module, (LongT5Attention, LongT5LocalAttention, LongT5TransientGlobalAttention)):
            # Mesh TensorFlow attention initialization to avoid scaling before softmax
            # See https://github.com/tensorflow/mesh/blob/fa19d69eafc9a482aff0b59ddd96b025c0cb207d/mesh_tensorflow/transformer/attention.py#L136
            d_model = self.config.d_model
            key_value_proj_dim = self.config.d_kv
            n_heads = self.config.num_heads
            module.q.weight.data.normal_(mean=0.0, std=factor * ((d_model * key_value_proj_dim) ** -0.5))
            module.k.weight.data.normal_(mean=0.0, std=factor * (d_model**-0.5))
            module.v.weight.data.normal_(mean=0.0, std=factor * (d_model**-0.5))
            module.o.weight.data.normal_(mean=0.0, std=factor * ((n_heads * key_value_proj_dim) ** -0.5))
            if module.has_relative_attention_bias:
                module.relative_attention_bias.weight.data.normal_(mean=0.0, std=factor * ((d_model) ** -0.5))
                if isinstance(module, LongT5TransientGlobalAttention):
                    module.global_relative_attention_bias.weight.data.normal_(
                        mean=0.0, std=factor * ((d_model) ** -0.5)
                    )
```
from t5 implementation https://github.com/huggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/models/longt5/modeling_longt5.py#L1291 

1)  we notice that the factor is multiplied by  ((d_model * key_value_proj_dim) ** -0.5)   for just the query and the output , and with * (d_model**-0.5) for key and value, why? Is there a detailed explanation of that? and still the initial value of the factor is 1.0?

2) Also today I found this issue https://github.com/huggingface/transformers/issues/16749

According to my understanding to this issue and correct me if I am wrong :

@patrickvonplaten  corrects the initialization but still vague for me is the relation between tying word embedding initialization and language model head initialization in this line https://github.com/huggingface/transformers/blob/d0acc9537829e7d067edbb791473bbceb2ecf056/src/transformers/models/t5/modeling_t5.py#L766 and why this condition in not included in longt5 implementation?


Comments:
It also confuses me.

@patrickvonplaten @patil-suraj @craffel

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Hey @Arij-Aladel,

The reason is that the first T5 model:
- https://huggingface.co/t5-base does tie the word embeddings where as the v1_1 version doesn't
- https://huggingface.co/google/t5-v1_1-base

=> therefore we need to support both use cases in Transfomers

@patrickvonplaten Is it the answer to initializing word embeddings(question2)? if yes, then sorry it is not clear yet. the condition clearly states that the initialization is only when there is no word tieing. So where is the support? what about the first question, please?

Sorry I won't have time to look into this anymore. Also note that the questions might find better answers in the forum: https://discuss.huggingface.co/

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Arij-Aladel I came here because I had exactly this same question as you - i.e. why are the weights initialized this way, and how does it avoid scaling the attention by $\frac{1}{\sqrt{features}}$ before softmax? 

Did you find the answer to this question? 

My guess is that the idea behind this is that during initialization if you multiply each weight by $\frac{1}{\sqrt{features}}$ then the factor affects the output of the layer, and then for the initialized weight the scaling isn't needed. However, it doesn't explain why the scaling isn't needed once the weights have been trained. i.e. in the original formulation, the scaling happens *all the time*, whereas here, the scaling happens only for the randomly initialized weights. 


This line: 

```
module.o.weight.data.normal_(mean=0.0, std=factor * ((n_heads * key_value_proj_dim) ** -0.5))
``` 

Is just confusing. It can be rewritten as: 

```
module.o.weight.data.normal_(mean=0.0, std=factor * (d_model ** -0.5))
``` 

That's because, for T5-small, 

```
  "d_ff": 2048,
  "d_kv": 64,
  "d_model": 512,
  "feed_forward_proj": "relu",
  "model_type": "t5",
  "num_heads": 8,
``` 

Now, `key_value_proj_dim = d_kv`, so `key_value_proj_dim = 64`. 

`n_heads * key_value_proj_dim` = `8 * 64` = `512`= `d_model`. 

I don't know why that line was written that way.