Bug when using StaticCache in Qwen2.5 Inference with custom inputs_embeds and attention_masks
Author: matthewdm0816
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.50.3
- Platform: Linux-6.8.0-52-generic-x86_64-with-glibc2.35
- Python version: 3.12.9
- Huggingface_hub version: 0.29.3
- Safetensors version: 0.5.3
- Accelerate version: 1.5.2
- Accelerate config:    - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: bf16
        - use_cpu: False
        - debug: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - enable_cpu_affinity: False
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
- DeepSpeed version: not installed
- PyTorch version (GPU?): 2.6.0+cu124 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: <fill in>
- GPU type: NVIDIA GeForce RTX 3090

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

When using Qwen2.5 models with custom `inputs_embeds` and `attention_mask` together with static KV cache implementation and flash attention 2, the generation fails with error:
```
File "/hd1/xxx/miniconda3/envs/vllm/lib/python3.12/site-packages/transformers/models/qwen2/modeling_qwen2.py", line 623, in _update_causal_mask
    raise ValueError(
ValueError: You are attempting to perform batched generation with padding_side='right' this may lead to unexpected behaviour for Flash Attention version of Qwen2. Make sure to  call `tokenizer.padding_side  = 'left'` before tokenizing the input.
```

The issue can be reproduced with the following minimal example:


```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

def reproduce_bug():
    # Load Qwen2.5 model
    model_name = "Qwen/Qwen2.5-1.5B-Instruct"
    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(
        model_name,
        torch_dtype=torch.bfloat16,
        attn_implementation="flash_attention_2",
    )

    model = model.cuda()

    tokenizer.padding_side = "left"  # Set padding to the left side
    model.generation_config.cache_implementation = "static"  # Use static cache
    model.generation_config.use_cache = True  # Enable KV cache
    
    text = "Hello, my name is"
    input_ids = tokenizer(text, return_tensors="pt").input_ids.to(model.device)
    
    # Get some inputs_embeds
    with torch.no_grad():
        inputs_embeds = model.get_input_embeddings()(input_ids)
    
    # Create custom attention_mask
    batch_size = 2
    seq_length = input_ids.shape[1]
    
    # Duplicate embeddings to create a batch
    batched_embeds = inputs_embeds.repeat(batch_size, 1, 1)
    
    # Create custom attention_mask
    attention_masks = torch.ones((batch_size, seq_length), dtype=torch.long, device=model.device)
    
    # Try to generate using custom inputs_embeds and attention_mask
    outputs = model.generate(
        inputs_embeds=batched_embeds,
        attention_mask=attention_masks,
        pad_token_id=tokenizer.pad_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

if __name__ == "__main__":
    reproduce_bug()
```

### Expected behavior

The model succeeds to generate.

Comments:
I found this issue may be related to https://github.com/huggingface/transformers/issues/34678#issue-2647279857

Moreover, I tested the `attn_implementation` arg, and found both `eager` and `sdpa` option works fine, but setting to `flash_attention_2` fails.

@matthewdm0816 It is the accepted behavior for flash attention. Pleaser refer the `_update_causal_mask` function in `qwen2_5_vl`

REF: https://github.com/huggingface/transformers/blob/800510c67bfc5cedd0bb7635648a07f39719be43/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L1232

Yeah, I don't think FA2 works with static cache

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.