Optimize flash attention max seqlen computation in vision attention
Author: ShareLer
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47170)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47170)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

This PR optimizes Flash Attention variable-length paths by precomputing the maximum sequence length once at the model/encoder level after `cu_seqlens` is built, then passing it down through blocks into each attention layer.

## Why

Previously, several vision/audio attention implementations recomputed:

```python
(cu_seqlens[1:] - cu_seqlens[:-1]).max()
```
inside every attention layer on every forward pass.
For Flash Attention varlen kernels, max_seqlen_q and max_seqlen_k are ultimately consumed as Python integer arguments. In eager mode, passing a 0-dimensional CUDA tensor can therefore move the scalar conversion to the Flash Attention / custom op boundary, which may introduce repeated device-to-host synchronization. For an N-layer encoder, this repeats the same max_seqlen computation and scalar conversion path in every layer.

## What changed
Compute max_seqlen once in the model/encoder forward path when Flash Attention is requested.
Compute max_window_seqlen once for sliding-window variants.
Pass the precomputed values down through blocks into attention modules.
Keep the attention-level fallback without .item() for standalone attention calls, preserving previous behavior when max_seqlen is not provided.
Guard the computation behind is_flash_attention_requested, so non-Flash-Attention paths do not pay this cost.

## Related
This is related to #44962 and the discussion in #44973. Instead of adding scalar conversion inside each attention layer, this PR lifts the computation to the model/encoder level where cu_seqlens is already available, reducing repeated per-layer work and avoiding repeated per-layer scalar conversions in the normal model forward path.


## Experiment
### Before
<img width="1341" height="496" alt="image" src="https://github.com/user-attachments/assets/6c5c6d0e-3cbe-451c-91fd-eca6988303ba" />
<img width="1478" height="355" alt="image" src="https://github.com/user-attachments/assets/e4e0ca92-1ed4-4255-a87f-0d960c5fdb1e" />
block kernel launch
<img width="1340" height="107" alt="image" src="https://github.com/user-attachments/assets/d3e88c10-65a8-439b-80b6-e198897e575e" />

### After
<img width="1025" height="568" alt="image" src="https://github.com/user-attachments/assets/ed73b0fc-d063-4b3a-b4d6-4d879906c815" />
<img width="1144" height="446" alt="image" src="https://github.com/user-attachments/assets/909a8461-fe37-44c3-929f-a92624873a15" />
<img width="1307" height="92" alt="image" src="https://github.com/user-attachments/assets/f846bdb2-baea-476f-84f6-79d7f496c556" />



## Code Agent Policy

The Transformers repo is currently being overwhelmed by a large number of PRs and issue comments written by
code agents. We are currently bottlenecked by our ability to review and respond to them. As a result, 
**we ask that new users do not submit pure code agent PRs** at this time. 
You may use code agents in drafting or to help you diagnose issues. We'd also ask autonomous "OpenClaw"-like agents
not to open any PRs or issues for the moment.

PRs that appear to be fully agent-written will probably be closed without review, and we may block users who do this
repeatedly or maliciously. 

This is a rapidly-evolving situation that's causing significant shockwaves in the open-source community. As a result, 
this policy is likely to be updated regularly in the near future. For more information, please read [`CONTRIBUTING.md`](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md).

- [x] I confirm that this is not a pure code agent PR.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://huggingface.co/docs/transformers/contributing) and the
      [Pull Request](https://huggingface.co/docs/transformers/pr_checks) checks?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes according to the [guidelines](https://github.com/huggingface/transformers/tree/main/docs)?
- [ ] Did you write any new necessary [tests](https://huggingface.co/docs/transformers/testing)?


## Who can review?
cc @vasqu

Comments:
> Imo totally valid and makes sense to avoid a per layer call
> 
> Only thing I would change design wise would be to move things to vision utils as well and make similar skips as when the kwarg already exists. cc @IlyasMoutawwakil @zucchini-nlp

Thanks for the suggestion! 
I moved the vision logic into `get_vision_max_seqlen` in `vision_utils.py` and added the same kwargs-based skip for precomputed values. And audio follows the existing model-local helper pattern.

> left a couple comments that should simplify it a bit, mostly nudging towards the same constrants we agreed upon in #45396:
> 
> * the vision util should abstract the repetitive if this is none
> * the "extra" arguments should stay hidden in kwargs

Thanks, addressed the review comments:
- `get_vision_max_seqlen` now accepts the model config, consumes a precomputed value from `kwargs` first, returns `None` for non-Flash-Attention paths otherwise, and computes a Python `int` once for Flash Attention.
- Added `get_vision_attention_seqlens` to return `cu_seqlens` and `max_seqlen` together.
- Kept the existing tensor-only `get_vision_cu_seqlens` API unchanged for backward compatibility.
- Removed `max_seqlen` from pass-through Block/Encoder signatures and kept it hidden in `**kwargs`; only the Attention modules that consume it declare it explicitly.
- Removed the Attention-level recomputation fallback. Flash Attention now raises a clear `ValueError` if `max_seqlen` was not precomputed.

> Seems like we just created a basic VisionRaggedInput data collator for qwen-style models. When we get the torch-ragged attn support, we might merge the input preparation as a single collating class?

One more for my data collator approach, I feel validated :laughing: :heart: 

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29552181954) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

> Super nice initiative, thanks! Cosmos3-Edge is close to merge, lets fix it as well as imo it'll be merged first
> 
> Seems like we just created a basic `VisionRaggedInput` data collator for qwen-style models. When we get the torch-ragged attn support, we might merge the input preparation as a single collating class?

Thanks for the review! I rebased onto the latest main and added Cosmos3-Edge to the same precomputation and fallback flow.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: cosmos3_edge, ernie4_5_vl_moe, exaone4_5, glm4v, glm4v_moe, glm_image, glm_ocr, hunyuan_vl, kimi_k25, minicpmv4_6, paddleocr_vl, qwen2_5_omni, qwen2_5_vl, qwen2_vl, qwen3_5, qwen3_5_moe

> Just some small comments left imo but this looks very solid now

Thanks! I’ve addressed the remaining comments and pushed the updates. Appreciate the thorough review!

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47170)
**Latest run:** [29625399091:2](https://github.com/huggingface/transformers/actions/runs/29625399091)
**Result:** `success` | **Jobs:** 15 | **Tests:** 172,162 | **Failures:** 0 | **Duration:** 15h 8m

<!-- ci-dashboard-recap:end -->