Very different output depending on whether an attention mask is passed when using caching
Author: naimenz
Pull request: no

### System Info

- `transformers` version: 4.42.0
- Platform: Linux-5.15.0-106-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.23.4
- Safetensors version: 0.4.3
- Accelerate version: 0.29.2
- Accelerate config:    not found
- PyTorch version (GPU?): 2.2.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: No
- Using GPU in script?: No
- GPU type: NVIDIA H100 80GB HBM3

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I am debugging issues I experienced when upgrading from `transformers==4.41.2` to `transformers==4.42.x`. Since `4.42.0`, there is a large difference in behavior when passing a cache based on whether you pass an attention mask or not.
```python
import torch
from transformers import AutoModelForSequenceClassification, AutoTokenizer

torch.manual_seed(0)
model_name = "EleutherAI/pythia-14m"
model = AutoModelForSequenceClassification.from_pretrained(model_name, use_cache=True)
tokenizer = AutoTokenizer.from_pretrained(model_name)

cache_inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
cache_outputs = model(**cache_inputs)
print(f"{cache_outputs.logits = }")
# On 4.41.2: tensor([[-1.0874,  1.5391]])
# On 4.42.0: tensor([[-1.0874,  1.5391]])

full_inputs = tokenizer(
    "Hello, my dog is cute. I like to eat ice cream.", return_tensors="pt"
)
len_cache = len(cache_outputs.past_key_values)
noncache_inputs = {
    "input_ids": full_inputs.input_ids[:, len_cache:],
    # "attention_mask": full_inputs.attention_mask,
}
full_outputs = model(**noncache_inputs, past_key_values=cache_outputs.past_key_values)
print(f"{full_outputs.logits = }")
# With attention mask:
# > On 4.41.2: tensor([[-2.9114,  0.3428]])
# > On 4.42.0: tensor([[-2.9107,  0.3432]])
# Without attention mask:
# > On 4.41.2: tensor([[-2.9114,  0.3428]]
# > On 4.42.0: tensor([[-1.7919,  0.4542]]
```

### Expected behavior

I expected the presence of an attention mask not to make a difference when there's no padding, as was the case in `4.41.2`. (I'm also a little surprised that there are small differences even with an attention mask, but they seem less bad currently.)

I'm happy to look into this more but thought I'd post it now in case it was obvious what the issue was.

Comments:
This is likely due to SDPA, can confirm the difference you're experiencing. You can temporarily switch back to the original attention implementation by loading the model like this ```model = AutoModelForSequenceClassification.from_pretrained(model_name, use_cache=True, attn_implementation="eager")```.

This is weird but it probably means that the default SDPA mask creation/usage is bugged somewhere when using a cache, since without it seems to work fine?

@naimenz I found the issue, it is indeed an issue with the mask creation or rather an issue of not even creating it. Will submit a PR in a bit and thanks for finding this as it most likely affects multiple models.

Amazing, thanks for looking into it so quickly!