Fix half-precision torch.compile crash in DETR-family sine position embeddings
Author: David-Wu1119
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=47238)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47238)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

Fixes #47228 (Sam3 bf16 compilation regression, v5.8.0 → v5.9.0).

**Root cause.** #45922 rewrote the sine position embedding to `torch.ones(..., dtype=torch.bool).cumsum(dim, dtype=dtype)`. That exact shape matches TorchInductor's `pointless_cumsum_replacement` post-grad pattern (`torch/_inductor/fx_passes/post_grad.py`), which rewrites `cumsum(full(...))` into an `arange`-based expression using the *ones tensor's* dtype — silently dropping the explicit `dtype=` argument. The embeddings come out `int64`, get promoted to `float32` by the normalize division, and the encoder q/k/v `addmm` crashes with `mat1 and mat2 must have the same dtype, but got Float and BFloat16`. Eager mode never runs the pass (hence eager works); the pre-#45922 formulation didn't match the pattern (hence v5.8.0 works). Bisect confirmation: `torch._inductor.config.pattern_matcher=False` makes v5.9.0+ pass.

**Fix.** Cast before the cumsum (`embed_mask = mask.to(dtype)` → `cumsum(dim)`), which restores the pre-#45922 numerics exactly and no longer matches the inductor pattern. The same construct was inherited by 13 other modeling files (sam2/sam3/edgetam families via modular from DETR; maskformer→mask2former/oneformer via copies), so the identical minimal patch is applied uniformly to all 14 (+ the deformable_detr modular source). The dtype-dropping pattern itself is arguably an inductor soundness bug — I'll file it upstream and link it here; regardless, transformers shouldn't emit the fragile construct in half precision.

**Verification** (Apple Silicon, torch 2.13, commit 0bc355418):
- Tiny randomly-initialized `Sam3Model` (mirroring `Sam3ModelTester` config, `HF_HUB_OFFLINE=1`): `torch.compile` forward in bf16 crashes with the reporter's exact error on both `cpu` and `mps` before this patch; passes on both after, with values matching eager.
- New regression test (`Sam3SinePositionEmbeddingTest::test_compiled_build_keeps_requested_dtype`): compiled build returns `bfloat16` — fails on the unpatched code (returns `float32`), passes with the fix.
- `tests/models/sam3` + `tests/models/detr` fast suites: identical results to `main` (same 7 pre-existing env-dependent failures on my machine, both sides; my new test passes).

## Before submitting
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md)?
- [x] Was this discussed via a GitHub issue? → #47228
- [x] Did you write any new necessary tests?

## Who can review?

@Cyrilvallez @molbap (vision / DETR-family; compile)

*Prepared with AI assistance; I reviewed and stand by every line, and reproduced the failure and fix locally before submitting.*

Comments:
Upstream inductor bug filed as promised: pytorch/pytorch#189518 (`pointless_cumsum_replacement` drops `cumsum`'s explicit `dtype` argument, with a 6-line standalone repro).

Hey! I like the idea, and I appreciate the workaround to torch's approach. Can we refactor this slightly, though?

1) Explicitly write the `arange` lines in the `if mask is None` block (basically inline the compiler optimization)
2) Only use `cumsum` in the `else` block

It's about the same amount of space, but it boosts eager performance a little. You can leave a comment mentioning the torch bug too, in case anyone is inclined to revert it too soon.

Done — exactly as suggested:

1. `mask is None` now builds `y_embed`/`x_embed` directly from `torch.arange(1, H+1)`/`arange(1, W+1)` (expanded), i.e. the compiler optimization written out — with the dtype requested explicitly, so it's immune to the inductor rewrite by construction.
2. `cumsum` only runs in the `else` branch for explicit masks.
3. The comment now points at the upstream bug (pytorch/pytorch#189518) with a note not to revert to `cumsum` until that fix is widely released.

Applied uniformly across the 14 files + the deformable modular source. Re-verified: `arange` output is exactly equal to `cumsum(ones)` (checked), the tiny-Sam3 bf16 compile repro passes on cpu and mps, the regression test passes, and the sam3/detr suites match main.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: conditional_detr, deformable_detr, detr, edgetam, edgetam_video, mask2former, maskformer, oneformer, sam2, sam2_video, sam3, sam3_lite_text, sam3_tracker_video

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=47238)
**Latest run:** [29286056908:2](https://github.com/huggingface/transformers/actions/runs/29286056908)
**Result:** `success` | **Jobs:** 2 | **Tests:** 19 | **Failures:** 0 | **Duration:** 2m 27s

<!-- ci-dashboard-recap:end -->

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_47238). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.