Fix inverted is_tracing condition causing FA2 long-context decode regression
Author: kamran-nizamani
Pull request: yes
Labels: name: Code agent slop

Fixes #46693

## What broke

In `_process_flash_attention_kwargs` (`src/transformers/modeling_flash_attention_utils.py`), `max_seqlen_q`/`max_seqlen_k` are converted from `torch.Tensor` to `int` via `.item()` so they can be passed to `flash_attn_varlen_func`. The conversion is guarded by `is_tracing(...)`:

```python
if not isinstance(max_seqlen_q, int) and is_tracing(max_seqlen_q):
    max_seqlen_q = max_seqlen_q.item()
```

This condition is inverted. `is_tracing()` returning `True` means we're under `torch.compile`/`torch.export`/CUDA graph capture — exactly the situation where we must **not** call `.item()`, since that forces a host sync and can graph-break dynamo (this is what the comment directly above the code describes). Instead, the code only converts to `int` when tracing, and leaves the value as a tensor in plain eager mode.

The practical effect in eager-mode decoding: the unconverted tensor is passed through to `flash_attn_varlen_func`, which performs its own internal sync to get a scalar, on every single decode step. `is_tracing()` itself isn't free either — it calls `is_torchdynamo_compiling()`, `is_jit_tracing()`, and `is_cuda_stream_capturing()` (a real `torch.cuda.is_current_stream_capturing()` call) on every step, for every attention layer.

For long-context generation, decode is hundreds/thousands of steps, each with a fixed per-step overhead added by this bug. Once the KV cache is large enough that the FA2 kernel itself is memory-bandwidth-bound, this fixed overhead is no longer negligible — which matches the reported regression profile (no regression with SDPA, which doesn't go through this path; no regression at small KV cache; ~17% slower at long context with FA2).

This was introduced in #42435 (FA4 initial support), which restructured this logic from an unconditional `.item()` call.

## The fix

Invert the condition: convert to `int` when **not** tracing, keep the tensor when tracing.

```python
if not isinstance(max_seqlen_q, int) and not is_tracing(max_seqlen_q):
    max_seqlen_q = max_seqlen_q.item()
```

Same fix applied to the `max_seqlen_k` branch. No other behavioral changes — this restores the original eager-mode semantics from before #42435 while keeping the new tracing-aware deferral that #42435 intended to add.

## Testing

- Confirmed the reproduction script from #46693 runs correctly with this change.
- Ran the existing FA2 attention test suite locally (no regressions).

Happy to add a dedicated regression test asserting `max_seqlen_q`/`max_seqlen_k` are plain `int` in eager mode if maintainers want one — let me know where you'd like it (e.g. `tests/utils/test_modeling_flash_attention_utils.py`).

cc @zucchini-nlp @Cyrilvallez @vasqu

Comments:
This PR was flagged by our automated quality checks. If you're a genuine
contributor, please reply here and a maintainer will review your PR.

Common reasons for flagging:
- New GitHub account
- Unusually high number of repository forks in a 24-hour window

We appreciate your contribution and apologize if this is a false positive!

This one is intentional because FA registers its fake ops with py ints so we cannot avoid this item call under compile. 

On recent torch versions, it at least doesnt cause a graph break anymore afaik. I still recommend to use our data collator with flattening that prepares all the necessary input without any of those head aches here

**CI Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46696)

Thank you for your contribution 🤗!

## CI Security Gate — automatic approval blocked

This PR was **not** automatically approved for CI because the security gate failed.

**Possible reasons:**
- The PR touches 50 or more files — only PRs with fewer than 50 changed files are automatically approved
- A changed file is outside the allowed directories (`src/`, `tests/`, `docs/`, `utils/`), has a disallowed extension (only `.py`, `.txt`, `.md` permitted outside `tests/` and `docs/`), or is not `.md`/`.yml` inside `docs/`
- A new high-severity security issue was detected in the changed Python files (Bandit check)

See the [workflow run](https://github.com/huggingface/transformers/actions/runs/29452639203) for the exact violations.

A maintainer can review and manually approve CI if a finding is a false positive.

We landed a fix in another PR #47134, so closing this