encoder decoder model compile failed after refactor cache
Author: jiqing-feng
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.55.0.dev0
- Platform: Linux-6.11.0-28-generic-x86_64-with-glibc2.35
- Python version: 3.11.13
- Huggingface_hub version: 0.34.2
- Safetensors version: 0.5.3
- Accelerate version: 1.8.1
- Accelerate config:    not found
- DeepSpeed version: not installed
- PyTorch version (accelerator?): 2.9.0.dev20250714+cpu (NA)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>


### Who can help?

@zucchini-nlp @ArthurZucker 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
import time
import requests
import torch
import PIL.Image
from transformers import pipeline

model_id = "nlpconnect/vit-gpt2-image-captioning"
image_to_text = pipeline("image-to-text", model=model_id, device="cpu", torch_dtype=torch.float16)
image_url = "https://ankur3107.github.io/assets/images/image-captioning-example.png"
image = PIL.Image.open(requests.get(image_url, stream=True, timeout=3000).raw)

for _ in range(10):
    output = image_to_text(image)

start = time.time()
output = image_to_text(image)
end = time.time()
print(f"eager mode pipeline latency {end - start}")

image_to_text.model.forward = torch.compile(image_to_text.model.forward)

for _ in range(10):
    output = image_to_text(image)

start = time.time()
output = image_to_text(image)
end = time.time()
print(f"compile mode pipeline latency {end - start}")
```

error log:
```
torch._dynamo.exc.TorchRuntimeError: Dynamo failed to run FX node with fake tensors: call_function <built-in function scaled_dot_pr
oduct_attention>(*(FakeTensor(..., size=(1, 12, 1, 64), dtype=torch.float16), FakeTensor(..., size=(1, 12, 394, 64), dtype=torch.fl
oat16), FakeTensor(..., size=(1, 12, 394, 64), dtype=torch.float16)), **{'attn_mask': FakeTensor(..., size=(1, 1, 1, 197), dtype=to
rch.float16), 'dropout_p': 0.0, 'scale': None, 'is_causal': False}): got RuntimeError('Attempting to broadcast a dimension of lengt
h 197 at -1! Mismatching argument at index 1 had torch.Size([1, 1, 1, 197]); but expected shape should be broadcastable to [1, 12,
1, 394]')
```

### Expected behavior

Before the PR [38635](https://github.com/huggingface/transformers/pull/38635), the script runs well and can get 1.5x speed-up.

Comments:
The issue comes from [_supports_default_dynamic_cache](https://github.com/huggingface/transformers/blob/main/src/transformers/generation/utils.py#L1973). It used to be `False` before the PR, but changed to `True` after the PR.

If the model is compilable when using the tuple cache instead of the dynamic cache, should we keep the default tuple cache before we enable all models to be compilable by the static cache?

cc @SunMarc  @Rocketknight1 

cc @zucchini-nlp this one looks a bit tricky

There is a bug in GPT-2 attention which wasn't noticed previously, when tuple cache was used. The encoder key/value states are being duplicated instead of re-used, and it needs a fix. I will submit a PR soon