A potential bug in ModuleUtilsMixin.get_extended_attention_mask
Author: jk-jung
Pull request: no

## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.13.0
- Platform: 
- Python version: 3.8.5
- PyTorch version (GPU?): 1.10.0+cu102
- Tensorflow version (GPU?):
- Using GPU in script?:
- Using distributed or parallel set-up in script?:

### Who can help
@LysandreJik 
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- ALBERT, BERT, XLM, DeBERTa, DeBERTa-v2, ELECTRA, MobileBert, SqueezeBert: @LysandreJik
- T5, BART, Marian, Pegasus, EncoderDecoder: @patrickvonplaten
- Blenderbot, MBART: @patil-suraj
- Longformer, Reformer, TransfoXL, XLNet, FNet, BigBird: @patrickvonplaten
- FSMT: @stas00
- Funnel: @sgugger
- GPT-2, GPT: @patrickvonplaten, @LysandreJik
- RAG, DPR: @patrickvonplaten, @lhoestq
- TensorFlow: @Rocketknight1
- JAX/Flax: @patil-suraj
- TAPAS, LayoutLM, LayoutLMv2, LUKE, ViT, BEiT, DEiT, DETR, CANINE: @NielsRogge
- GPT-Neo, GPT-J, CLIP: @patil-suraj
- Wav2Vec2, HuBERT, SpeechEncoderDecoder, UniSpeech, UniSpeechSAT, SEW, SEW-D, Speech2Text: @patrickvonplaten, @anton-l

If the model isn't in the list, ping @LysandreJik who will redirect you to the correct contributor.

Library:

- Benchmarks: @patrickvonplaten
- Deepspeed: @stas00
- Ray/raytune: @richardliaw, @amogkam
- Text generation: @patrickvonplaten @narsil
- Tokenizers: @LysandreJik
- Trainer: @sgugger
- Pipelines: @Narsil
- Speech: @patrickvonplaten, @anton-l
- Vision: @NielsRogge, @sgugger

Documentation: @sgugger

Model hub:

- for issues with a model, report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj

For research projetcs, please ping the contributor directly. For example, on the following projects:

- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

## Information

Model I am using (Bert, XLNet ...): T5

There is a potential bug in ModuleUtilsMixin.get_extended_attention_mask, and actually, it has happened to me while I've trained the T5 model from scratch. In the function, it masks a tensor by setting a large negative number(-1e-4), since it will be added to the raw scores before the softmax. 
However, occasionally the value -1e4 is not small enough to nullify the scores in masked positions. In my case, some values in the raw scores before the softmax were small then -1e4 during training, so the model couldn't be trained correctly.

Here is the code I mentioned: [link](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L295-L302)

I think you use -1e4 because of fp16 compatibility, then how about dividing the case based on dtype like in [the code](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_utils.py#L233-L236).


## To reproduce


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

## Expected behavior

the function get_extended_attention_mask uses the smaller number to mask a tensor.

<!-- A clear and concise description of what you would expect to happen. -->


Comments:
I understand that the choice of 1e-4 may be suboptimal and that the masking may not be enough. I believe we've had this conversation before but I can't find it anywhere, I'm open to revisiting it.

WDYT @patrickvonplaten @sgugger @patil-suraj @stas00 

I agree 1e-4 might not be the best choice for every model. I think it's a good idea to handle this based on the `dtype`

Also, I think maybe we should add an argument `masked_value` to `get_extended_attention_mask` to specify the default mask value for each model from its original implementation.

@LysandreJik related discussion #10484

What @patil-suraj said, we for example already do it here:

https://github.com/huggingface/transformers/blob/033c3ed95a14b58f5a657f5124bc5988e4109c9f/src/transformers/modeling_utils.py#L233-L236

I'm also curious what you guys think is the best move here.
IMO, the -10e4 comes from the original Google implementation of BERT and we just copied it everywhere. However I've now seen a couple of issues related to this. In addition to the one @patil-suraj posted we also have this one https://github.com/huggingface/transformers/issues/14521#issuecomment-992428858

@LysandreJik  @sgugger @stas00 @patil-suraj - do you think it could make sense to change -10e4 to the minimum value of that dtype? 

Makes sense to me!

Could it be that the original implementation had this hardcoded because it worked with a single specific dtype?

I think it makes a total sense to use the logic in https://github.com/huggingface/transformers/issues/14859#issuecomment-998969055

I think leveraging the minimum value for the `dtype` as @patrickvonplaten would work too and be quite better than the if/else statement, right? Something like `torch.finfo(self.dtype).min`

> `torch.finfo(self.dtype).min`

Loving it! 

We should do the same for all other places we have the conditional for when we that for masking and where no weights will be impacted by this change.


Ok, happy to do a PR for this next week - putting it on my ToDo list

Centralizing all discussions on this:

https://github.com/huggingface/transformers/issues/9594
https://github.com/huggingface/transformers/issues/10484
https://github.com/huggingface/transformers/issues/15199

For people using GPT2 Model, there is a hacked hot fix of this issue:

```python
for name, buffer in model.named_buffers():
    if '.masked_bias' in name:
        buffer.data = torch.tensor(float('-inf'))
```

Add these codes after model initialization.

I won't be able to look into this until the next 3,4 weeks - still have it on my ToDo list though. Depending on how important this is, feel free to take over whoever is interested

Any ideas which `transformers` versions are affected?

We've always used -10_000 from the very beginning for the `attention_mask` so changing this now would affect all transformers versions

Hi, this is on my TODO list - I have a few remaining things to finalize about PT/TF/Flax more aggressive testings, and will come back to this issue!

Start working on it :-)

@jk-jung 

This is fixed now (finally) :-)

When i install version 4.29.0.dev0, there is a bug in modeling_utils.py in line902:
TypeError: torch.iinfo() requires an integer input type. Use torch.finfo to handle 'torch.iinfo'
Could you plz give some advices? @ydshieh 

@FayeXXX Could you show us a short code snippet to reproduce the issue please? Thank you!