Self attention in T5 decoder does not work as expected
Author: Arij-Aladel
Pull request: no

Hello! 
I have been struggling with this problem for a while. I was trying to train T5 using just 2 layers in the encoder and 2 layers in the decoder from scratch on MLM task. Drawing self-attention of the decoder I saw that the attention weights from the first epoch are not masked properly. where the max_seq_length of the masked sequence is 128. I have trained this [tokenizer](https://github.com/huggingface/transformers/tree/master/examples/flax/language-modeling#train-tokenizer-2) on wiki-103 text and HotpotQA text then I built a masking PyTorch pipeline following the paper steps depending on this [script](https://github.com/huggingface/transformers/blob/master/examples/flax/language-modeling/run_t5_mlm_flax.py). This [repo](https://github.com/Arij-Aladel/T5-Tasks) contains the reproducible code and this [folder](https://github.com/huggingface/transformers/issues/15734#issue-1144808323) contains the processed data and the checkpoints.  To see the problem with self-attention masks in decoder refer please to this [notebook](https://github.com/Arij-Aladel/T5-Tasks/blob/main/T5-heatmap_MLM_test_128.ipynb)

Comments:
@patrickvonplaten @patil-suraj @sgugger @stas00 

I have tracked the mask it was generated normally. I even tried to decrease the[ masking value ](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/src/transformers/modeling_utils.py#L307)from -10000 to -10000000000 . The issue is still. I do not know where is the problem, but apparently, it appears after summing the mask to position bias and scores and finally applying the softmax [here](https://github.com/huggingface/transformers/blob/db7d6a80e82d66127b2a44b6e3382969fdc8b207/src/transformers/models/t5/modeling_t5.py#L513). 

Note: I thought it is an overfitting problem so I increased the number of layers gradually to be (4:4=encoder:decoder, 6:6=encoder: decoder). That mitigates this problem but the problem still exists.

HI @Arij-Aladel it would be awesome if you could post a short code snippet without much custom code here so we can reproduce easily. Also for general questions like this please use the [forum](https://discuss.huggingface.co/) as we issues for bug reports and feature requests. Thanks!

@patil-suraj  Thanks for your reply. It is not a general question it is a very specific question related to self-attention masking in the T5 decoder. No need to verify my code. The issue in T5 masking in the attention in the decoder, it does not work properly and I showed an example in the [notebook](https://github.com/Arij-Aladel/T5-Tasks/blob/main/T5-heatmap_MLM_test_128.ipynb).

@patrickvonplaten @patil-suraj @sgugger @stas00
It is easy to reproduce the results using the repo all steps are listed to reproduce the results. Just I want to know the reason that weights in causal self attention in decoder is not masked properly.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

https://github.com/huggingface/transformers/commit/ece762443e4269e50cc1a348838e63e4a499575d  this explains alot  thanks for fixing the problem