🚨 Enable SDPA (and other attention backends) for T5 and propagate to the T5 family
Author: jiqing-feng
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47014)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47014)
<!-- ci-dashboard-badge:end -->

## What this PR does

Refactors the **T5** attention stack to route through `ALL_ATTENTION_FUNCTIONS`,
so T5 can dispatch to `sdpa` / `eager` instead of the old eager-only path, and
propagates the change to the copied-from family.

Now `_supports_sdpa = True` / `_supports_attention_backend = True` and running
`test_eager_matches_sdpa_inference`:
- `t5` (reference), `mt5`, `udop`, `pop2piano`
- `pix2struct` — also required migrating the vision-tower attention
  (`Pix2StructVisionAttention`), a standard dense attention that was not on the
  interface, otherwise SDPA is unusable for the full model.
- `umt5` — its attention is its own dense implementation (not `# Copied from`
  T5); migrated the same way.

`switch_transformers` (modular, inherits T5 attention) is regenerated so its
modeling matches the refactored T5, but it stays eager-only (`_supports_sdpa`
not set): it is not on the SDPA path, this only propagates the eager refactor.

Alignment applied to migrated modules: attention goes through
`ALL_ATTENTION_FUNCTIONS` (with `eager_attention_forward` fallback), `self.scaling`,
`is_causal`; the relative position bias is folded into the additive attention mask;
forwards use `**kwargs: Unpack[TransformersKwargs]` + `@can_return_tuple` /
`@auto_docstring` / `@merge_with_config_defaults` / `@capture_outputs`, dropping the
manual `return_dict` / `output_attentions` / `output_hidden_states` resolution.

## 🚨 Behavior changes

- With `sdpa`, softmax no longer force-upcasts to fp32 as the old path did;
  numerics stay within the `eager_matches_sdpa` tolerance.
- `output_attentions` removed from internal block/layer tuple returns; attentions
  are collected via `capture_outputs`.

`test_eager_matches_sdpa_inference` is not skipped for any enabled model.

## longt5 stays not-SDPA-enabled

The regular `LongT5Attention` (copied from T5) is on the interface, but the encoder
always runs the block-sparse `LongT5LocalAttention` / `LongT5TransientGlobalAttention`.
These operate on 5D blocked tensors `(batch, num_blocks, heads, block_len, 3*block_len)`,
while `sdpa_attention_forward` assumes 4D `(batch, heads, seq, dim)` — they cannot go
through the interface at all. `LongT5EncoderModel` is purely block-sparse, so the
framework refuses to switch its attention implementation:

> LongT5EncoderModel does not support setting its attention implementation
> dynamically, because it does not follow the functional approach based on
> AttentionInterface

Forcing `_supports_sdpa = True` makes the encoder-only sdpa tests fail (fp16
`mean relative difference: nan`). Same precedent as `pegasus_x` (block-sparse
encoder + regular decoder), which also sets `_supports_sdpa = False`.

## Testing

`utils/check_copies.py` passes; full sweep green:

```
tests/models/{t5,mt5,umt5,udop,pop2piano,longt5,pix2struct,switch_transformers}/test_modeling_*.py
-> 1817 passed, 1958 skipped, 18 xfailed, 6 xpassed, 16699 subtests passed, 0 failed
```



Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: longt5, mt5, pix2struct, pop2piano, switch_transformers, t5, udop, umt5

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47014)
**Latest run:** [28567574479:2](https://github.com/huggingface/transformers/actions/runs/28569076651)
**Result:** `success` | **Jobs:** 2 | **Tests:** 9 | **Failures:** 2 | **Duration:** 51s

<!-- ci-dashboard-recap:end -->

Attention dispatch so cc @arthurzucker @cyrilvallez

WIll check it out when I have time but it's a big PR so expect delays :pray: 

This competes with https://github.com/huggingface/transformers/pull/46946

FWIW, we might take advantage of recent updates to accommodate Inkling, and support FlexAttn with position bias. I didn't check in details, might be wrong if these two `position-biases` are added up in different ways

https://github.com/huggingface/transformers/blob/290d5c4c1fc72c4c32aea6abd79c8809dee01462/src/transformers/models/inkling/modeling_inkling.py#L602-L609

You could only do this with flex attention using score mod afaik, but flex attention with score mod was quite slow; things might have changed by then but their mask creation is the biggest blocker to make flex attention viable.

Edit: This might be ok when triggered via FA4 but still fear the mask creation tbh

Ok yea checked, we added it to the score mod. We can do it for sure, just doubting the perf a bit
https://github.com/huggingface/transformers/blob/290d5c4c1fc72c4c32aea6abd79c8809dee01462/src/transformers/integrations/flex_attention.py#L271