[qwen2-vl] fix FA2 inference
Author: zucchini-nlp
Pull request: yes
Labels: name: for patch

# What does this PR do?

Fixes https://github.com/huggingface/transformers/issues/39095 and set `mask=None` for FA2, otherwise inference fails because FA2 expects a 2d mask

Tested with `RUN_SLOW=1 pytest -k flash_attn` for all modified models

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_39121). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

I don't think there was a specific reason, it has been that way since the first model was released. In any case Qwen-VL models seem to be based on FA2 first, while other attn are added just because we wanted to support them

Thus the mask is prepared based on `cu_seqlens`

@zucchini-nlp  Hi, the FA2 may not be working as expected. This is because we use the following code in `_flash_attention_forward`:

https://github.com/huggingface/transformers/blob/ed36f8490eb3748f2424d854936cc6f816aeb486/src/transformers/modeling_flash_attention_utils.py#L423-L426

However, there is an issue with the variable names here, which causes  FA2 to not function correctly.

https://github.com/zucchini-nlp/transformers/blob/2cb7f21991478494c963ab8c4b52691d8d192b0d/src/transformers/models/qwen2_5_vl/modeling_qwen2_5_vl.py#L266-L269

After correcting the code to the following version, I tested the model accuracy and it worked as expected 🤗

```python
        attn_output, _ = attention_interface(
            self,
            query_states,
            key_states,
            value_states,
            attention_mask=attention_mask,
            dropout=0.0,
            scaling=self.scaling,
            cu_seq_lens_k=cu_seqlens,  # pass cu seq lens for FA2
            cu_seq_lens_q=cu_seqlens,
            max_length_q=max_seqlen,
            max_length_k=max_seqlen,
            is_causal=False,
            **kwargs,
        )
```

@JJJYmmm yeah, thanks, the current state doesn't work yet. In fact renaming only doesn't help because the `varlen` codepath needs position ids as well. I have the local version working and now I am trying to find the reason to a linked issue. Even though it's not caused by recent changes, we need to fix it 



> @JJJYmmm yeah, thanks, the current state doesn't work yet. In fact renaming only doesn't help because the `varlen` codepath needs position ids as well. I have the local version working and now I am trying to find the reason to a linked issue. Even though it's not caused by recent changes, we need to fix it

Oh! Yes, I also modified the logic in `_flash_attention_forward` — when `position_ids` is `None` but `cu_seqlens` is not `None`, it now directly calls the varlen func. Looking forward to your changes!

@vasqu @JJJYmmm I updated the fa2 path so it uses `flash_attn_with_varlen` for pad-free inputs. Can you check once again and if everything is fine, I'll merge

@zucchini-nlp  LGTM 🫡

@zucchini-nlp can we refactor the attention mask creation to be outside, cu seqs are not different for a layer!

![image](https://github.com/user-attachments/assets/8c98f094-7fa7-4b43-91d7-2f280e2022a2)
Actually on cuda it makes a somewhat 2x diff