kernels-community/flash-attn3 does not work with SmolVLM2
Author: MilkClouds
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 5.0.0.dev0
- Platform: Linux-5.15.0-161-generic-x86_64-with-glibc2.35
- Python version: 3.11.14
- Huggingface_hub version: 1.1.2
- Safetensors version: 0.6.2
- Accelerate version: 1.11.0
- Accelerate config:    not found
- DeepSpeed version: 0.17.5
- PyTorch version (accelerator?): 2.8.0+cu128 (CUDA)
- Using distributed or parallel set-up in script?: no
- Using GPU in script?: yes
- GPU type: NVIDIA A100 80GB PCIe

### Who can help?

@MekkCyber @zucchini-nlp 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```py
import torch

from transformers import AutoModelForImageTextToText, AutoProcessor


model_path = "HuggingFaceTB/SmolVLM2-256M-Video-Instruct"
processor = AutoProcessor.from_pretrained(model_path)
model = AutoModelForImageTextToText.from_pretrained(
    model_path, torch_dtype=torch.bfloat16, _attn_implementation="kernels-community/flash-attn3"
).to("cuda")

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/bee.jpg",
            },
            {"type": "text", "text": "Can you describe this image?"},
        ],
    },
]

inputs = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=True,
    return_dict=True,
    return_tensors="pt",
).to(model.device, dtype=torch.bfloat16)

generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)
generated_texts = processor.batch_decode(
    generated_ids,
    skip_special_tokens=True,
)
print(generated_texts[0])

```

### Expected behavior

fa3 must work but it crash.

```sh
$ srun --nodes 1 --gpus 1 --cpus-per-gpu 16 --mem 16G --pty --partition a100 --nodelist=A100-Bumblebee python test_smolvlm.py               ✭
Using a slow image processor as `use_fast` is unset and a slow processor was saved with this model. `use_fast=True` will be the default behavior in v4.52, even if the model was saved with a slow processor. This will result in minor differences in outputs. You'll still be able to use a slow processor with `use_fast=False`.
`torch_dtype` is deprecated! Use `dtype` instead!
Fetching 4 files: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 4129.27it/s]
Fetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 69615.00it/s]
Fetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 83468.74it/s]
Download complete: : 0.00B [00:00, ?B/s]                                                                                                                                                                                             | 0/4 [00:00<?, ?it/s]
Download complete: : 0.00B [00:00, ?B/s] [00:00, ?B/s]
Fetching 4 files: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:00<00:00, 39199.10it/s]
Download complete: : 0.00B [00:00, ?B/s] [00:00, ?B/s]
Download complete: : 0.00B [00:00, ?B/s]                                                                                                                                                                                             | 0/4 [00:00<?, ?it/s]
Traceback (most recent call last):
  File "/mnt/home/claude/GitHub/transformers/test_smolvlm.py", line 33, in <module>
    generated_ids = model.generate(**inputs, do_sample=False, max_new_tokens=64)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 120, in decorate_context
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/generation/utils.py", line 2673, in generate
    result = decoding_method(
             ^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/generation/utils.py", line 2866, in _sample
    outputs = self._prefill(input_ids, generation_config, model_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/generation/utils.py", line 3842, in _prefill
    return self(**model_inputs, return_dict=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/utils/generic.py", line 757, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 895, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/utils/generic.py", line 757, in wrapper
    output = func(self, *args, **kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 705, in forward
    image_hidden_states = self.get_image_features(pixel_values, pixel_attention_mask).to(inputs_embeds.device)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 631, in get_image_features
    image_hidden_states = self.vision_model(pixel_values=pixel_values, patch_attention_mask=patch_attention_mask)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/utils/generic.py", line 927, in wrapper
    outputs = func(self, *args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 407, in forward
    encoder_outputs: BaseModelOutput = self.encoder(
                                       ^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 336, in forward
    layer_outputs = encoder_layer(
                    ^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/modeling_layers.py", line 94, in __call__
    return super().__call__(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 297, in forward
    hidden_states, _ = self.self_attn(
                       ^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/models/smolvlm/modeling_smolvlm.py", line 246, in forward
    attn_output, attn_weights = attention_interface(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/integrations/flash_attention.py", line 70, in flash_attention_forward
    attn_output = _flash_attention_forward(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/GitHub/transformers/src/transformers/modeling_flash_attention_utils.py", line 616, in _flash_attention_forward
    out_unpad = flash_varlen_fn(
                ^^^^^^^^^^^^^^^^
  File "/mnt/raid12/cache/huggingface/hub/models--kernels-community--flash-attn3/snapshots/cdde5f42030606c40df258eebc1d73216b117e93/build/torch28-cxx11-cu128-x86_64-linux/flash_attn3/flash_attn_interface.py", line 597, in flash_attn_varlen_func
    return FlashAttnVarlenFunc.apply(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/autograd/function.py", line 576, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/raid12/cache/huggingface/hub/models--kernels-community--flash-attn3/snapshots/cdde5f42030606c40df258eebc1d73216b117e93/build/torch28-cxx11-cu128-x86_64-linux/flash_attn3/flash_attn_interface.py", line 363, in forward
    out, softmax_lse, *rest = _flash_attn_forward(
                              ^^^^^^^^^^^^^^^^^^^^
  File "/mnt/raid12/cache/huggingface/hub/models--kernels-community--flash-attn3/snapshots/cdde5f42030606c40df258eebc1d73216b117e93/build/torch28-cxx11-cu128-x86_64-linux/flash_attn3/flash_attn_interface.py", line 59, in _flash_attn_forward
    out, softmax_lse, *rest = flash_attn_3_cuda.fwd(
                              ^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home/claude/miniforge3/envs/owa/lib/python3.11/site-packages/torch/_ops.py", line 1243, in __call__
    return self._op(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: cu_seqlens_q must have shape (batch_size + 1)
srun: error: A100-Bumblebee: task 0: Exited with exit code 1
```

with simple patches as follow I can fix the issue but not sure it's the best, intended solution.

place: https://github.com/huggingface/transformers/blob/1c36d407d5ce0712c702e4314c62705c012bb700/src/transformers/models/smolvlm/modeling_smolvlm.py#L398-L405

fix:
<img width="1844" height="430" alt="Image" src="https://github.com/user-attachments/assets/829716b5-bb02-47e4-8acb-40954a213149" />


Also there's some more place with `if self.config._attn_implementation != "flash_attention_2":` across transformers and I guess all models would not work with same reason.

Comments:
Oh yeah, the attn implementation was renamed to simply `flash_attention` for both FA2 and FA3. So we need to check if `flash_attention in self.config.attn_implementation` to support current models and be backwards compatible with remote-code models

Feel free to open a PR and fix it everywhere

I ran a grep command to check which models have the same issue 
```bash
$ grep -r 'if self.config._attn_implementation != "flash_attention_2":' src/transformers/models/
```
<details>

```text
src/transformers/models/x_clip/modeling_x_clip.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/idefics3/modeling_idefics3.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/altclip/modeling_altclip.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/idefics2/modeling_idefics2.py:        elif self.config._attn_implementation != "flash_attention_2":
src/transformers/models/clipseg/modeling_clipseg.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/idefics/vision.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/git/modeling_git.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/smolvlm/modeling_smolvlm.py:        if self.config._attn_implementation != "flash_attention_2":
src/transformers/models/kosmos2/modeling_kosmos2.py:        if self.config._attn_implementation != "flash_attention_2":
```
</details>

oh nvm there's a lot more 
```bash
$ grep -r 'if self.config._attn_implementation == "flash_attention_2":' src/transformers/models/
```


<details>

```text
src/transformers/models/longt5/modeling_longt5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/falcon_h1/modeling_falcon_h1.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/falcon_h1/modular_falcon_h1.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/udop/modeling_udop.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_vl/modeling_qwen2_vl.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_5_omni/modular_qwen2_5_omni.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/gpt_neox_japanese/modeling_gpt_neox_japanese.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/musicgen_melody/modeling_musicgen_melody.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/kyutai_speech_to_text/modeling_kyutai_speech_to_text.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/video_llama_3/modeling_video_llama_3.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/video_llama_3/modular_video_llama_3.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/pop2piano/modeling_pop2piano.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen3_omni_moe/modeling_qwen3_omni_moe.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen3_omni_moe/modeling_qwen3_omni_moe.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/gptj/modeling_gptj.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/nemotron/modeling_nemotron.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/persimmon/modeling_persimmon.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/gpt_bigcode/modeling_gpt_bigcode.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/stablelm/modeling_stablelm.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen3_vl_moe/modeling_qwen3_vl_moe.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/glm4v/modeling_glm4v.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/zamba2/modeling_zamba2.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moonshine/modular_moonshine.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moonshine/modular_moonshine.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moonshine/modeling_moonshine.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moonshine/modeling_moonshine.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen3_vl/modeling_qwen3_vl.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/pixtral/modeling_pixtral.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/pixtral/modeling_pixtral.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/pix2struct/modeling_pix2struct.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/bloom/modeling_bloom.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modeling_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modeling_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modeling_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modeling_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modular_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modular_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modular_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/modernbert/modular_modernbert.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/sew/modeling_sew.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/sew/modular_sew.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moshi/modeling_moshi.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/moshi/modeling_moshi.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/kosmos2_5/modeling_kosmos2_5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/kosmos2_5/modeling_kosmos2_5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/t5/modeling_t5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/musicgen/modeling_musicgen.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/musicgen/modeling_musicgen.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/mt5/modeling_mt5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/falcon/modeling_falcon.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/bamba/modular_bamba.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/bamba/modeling_bamba.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/switch_transformers/modular_switch_transformers.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/switch_transformers/modeling_switch_transformers.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/bark/modeling_bark.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/bark/modeling_bark.py:            if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/umt5/modeling_umt5.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/glm4v_moe/modeling_glm4v_moe.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/zamba/modeling_zamba.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/codegen/modeling_codegen.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/opt/modeling_opt.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/mllama/modeling_mllama.py:        if self.config._attn_implementation == "flash_attention_2":
src/transformers/models/gpt_neo/modeling_gpt_neo.py:        if self.config._attn_implementation == "flash_attention_2":
```
</details>
I guess a lot of the models might be inheriting stuff but these are all the ones that have the issue .



`kernels-community/vllm-flash-attn3` is the value for `self.config_attn_implementation` . 
```python
if "flash" in self.config._attn_implementation:
            if not torch.any(~patch_attention_mask):
                patch_attention_mask = None
        else:
            print(self.config._attn_implementation)
            patch_attention_mask = _prepare_4d_attention_mask(patch_attention_mask, hidden_states.dtype)
```
Is this the correct fix ? @zucchini-nlp 


Yes we need to check for "flash" in ... to also support the kernels. Ideally, we should switch to https://github.com/huggingface/transformers/blob/dba6aeb1e389416ebaae29f2351af71cdd6c82ac/src/transformers/masking_utils.py#L896
This will be more flexible / easier to maintain (+supports all the implementations out of the box).

```python
patch_attention_mask = create_bidirectional_mask(
            config=self.config,
            input_embeds=hidden_states,
            attention_mask=patch_attention_mask,
        )
```
okay so i've replaced the entire if block with this and it seems to be working correctly . 
Should I go through like each model file and replace it with this? Or is there a better way?
And what's the command to generate the modeling files after the changes are made? 

You can generate the modeling file from modular via `python utils/modular_model_converter.py your_model`.

Imo, yes we should as that is the proper way to create masks moving forward - would like to have confirmation from @zucchini-nlp if she is ok with this tho.

Yes, sounds good to me as long as the tests pass!

Alright I'll start working on it

What about causal models like altclip?

Not directly related to this issue, but fixes there are also welcome! Essentially any model that still does not use `create_causal_mask` and/or `create_bidirectional_mask` should be fixed to use these instead

In a lot of the models like for example in `src/transformers/models/idefics/vision.py`
```python
if self.config._attn_implementation != "flash_attention_2":
    if attention_mask is not None and causal_attention_mask is not None:
        attention_mask = attention_mask + causal_attention_mask
    elif causal_attention_mask is not None:
        attention_mask = causal_attention_mask
else:
    self.is_causal = causal_attention_mask is not None
``` 
this is the current implementation , can we just replace this entire block with `create_causal_mask` ? 

In `src/transformers/models/modernbert/modeling_modernbert.py` can i just replace the if conditional with `if "flash" in self.config._attn_implementation` or is there a different fix
Current one 
```python
if self.config._attn_implementation == "flash_attention_2":
            if indices is None and cu_seqlens is None and max_seqlen is None:
                if batch_size is None and seq_len is None:
                    if inputs_embeds is not None:
                        batch_size, seq_len = inputs_embeds.shape[:2]
                    else:
                        batch_size, seq_len = input_ids.shape[:2]
                device = input_ids.device if input_ids is not None else inputs_embeds.device
```
And if I'm not wrong pretty much all the models that contain "flash_attention_2" need to be patched right? 

> And if I'm not wrong pretty much all the models that contain "flash_attention_2" need to be patched right?

Yes, otherwise it won't work with FA2 and FA3.


In `src/transformers/models/idefics/vision.py` there seems to be a bad copy and the attention mask was copied from CLIP-like models which wasn't designed well in the first place. Since it is a vision-only model. it supports bidirectional mask and not causal. Right now none of the two masks is being passed so the whole block is a no-op.Would be really nice to fix it by setting `is_causal = False` and preparing a bidirectional mask

Most vision-models have issues with attention mask and it's not standard yet...

It's always a case by case basis sadly and a lot of bad copied code. So it's hard to tell sometimes if you only need a bidirectional mask, causal mask etc - it has to be seen from the context of the initial mask creation. Clip also is a special case where parts are causal and others were not (#41750) which adds to the confusion.

The easiest fix is to replace all "flash_attention_2" stuff with "flash" (this enables also kernels fa versions) but tbh we would like to properly use our new mask API instead - it's more robust and easier to fix when stuff doesn't work.

Alright thanks for the info . 
>So it's hard to tell sometimes if you only need a bidirectional mask, causal mask etc

Yeah I can't really tell tbh . 

>we would like to properly use our new mask API instead - it's more robust and easier to fix when stuff doesn't work.

Sounds good I'll look into it