Suggestion: Give a warning for 4D attention mask which use float dtype but in 0.0 and 1.0 term (instead of -inf and zero).
Author: AbdiHaryadi
Pull request: no
Labels: name: Feature request

### Feature request

The title said it all.

### Motivation

Let's say I have a code like this: (Early warning: If you run this code, it will give an error.)

```py
import torch
from transformers import AutoModel

model = AutoModel.from_pretrained("google-bert/bert-base-uncased", device_map="auto")
attention_mask = torch.tensor([
    [1, 0],
    [0, 0],
], device=model.device).unsqueeze(0).unsqueeze(1)

outputs = model(
    input_ids=torch.tensor([[200, 201]], device=model.device),
    attention_mask=attention_mask,
)
print("Output 1:")
print(outputs.last_hidden_state[0, 0, :10])

outputs = model(
    input_ids=torch.tensor([[200, 202]], device=model.device),
    attention_mask=attention_mask,
)
print("Output 2:")
print(outputs.last_hidden_state[0, 0, :10])
```

Notice that the last input ID for output 1 and output 2 are different: 201 and 202. However, since it's designed to be masked, then the output should not be impacted.

But if you run that code, it will return an error because the attention mask are expected to be in float or boolean type.

---

My previous attempt was converting it to float, so the line 5-8 should be changed like this:
```py
...
attention_mask = torch.tensor([
    [1, 0],
    [0, 0],
], device=model.device).unsqueeze(0).unsqueeze(1).float()
...
```

But here's the funky part: if you use that change, both outputs give different result! I was surprising when I found it, so I try to debug it.

After a debug session that leads me to a [Pytorch page](https://docs.pytorch.org/docs/2.12/generated/torch.nn.functional.scaled_dot_product_attention.html), my mistake is that it should be changed to boolean, so the change should be like this:
```py
...
attention_mask = torch.tensor([
    [1, 0],
    [0, 0],
], device=model.device).unsqueeze(0).unsqueeze(1).bool()
...
```

If I want to use in a float term, the 0 terms should be changed to negative infinity, and the 1 terms should be changed to zero. This is because the attention mask is treated as bias with addition operator.

There is no warning for this, so careless developer like me can just use the mask without knowing the side effect.

### Your contribution

I'm not sure if I can make a PR for this case because I haven't explored how to apply the warning in general. What I found right now is just a specific case, and I hope it can help.

Comments:
This looks like a tricky issue. I think the cause is that we have attention mask preprocessing code (`_preprocess_mask_arguments`) but this is skipped if a 4D attention mask is passed, because the code assumes it's already prepared:

```python
# If the mask is already 4D, simply return as-is (it was already prepared, or it is custom)
  if isinstance(attention_mask, (torch.Tensor, BlockMask)) and len(attention_mask.shape) == 4:
      return True, attention_mask, None, None, None, None, None
```

Passing to core maintainers @ArthurZucker @cyrilvallez @vasqu since this is really a question about how our core modeling code should handle this case

It might be more of a documentation issue / it is not super obvious
- https://huggingface.co/docs/transformers/v5.10.2/en/glossary#attention-mask
- https://github.com/huggingface/transformers/blob/cbb65a4815d44f1d8b8ff7f51cca24ce491fc09e/src/transformers/utils/auto_docstring.py#L1880-L1890

We always expect 2D masks (torch int/long/bool). It was only recently that we properly added support 4D attention masks BUT on the condition that the user knows what he does; the documentation is lacking here (as for many power features 😢). I don't think we should really change this on our side but moreso maybe clarify. In any case also cc @stevhliu