Remove unnecessary expand_as in get_placeholder_mask across VLMs
Author: syncdoth
Pull request: yes

Fixes #44906

## Summary

- Remove `.expand_as(inputs_embeds)` from placeholder mask creation in `get_placeholder_mask` and equivalent inline patterns across all VLM models. `masked_scatter` natively broadcasts `(B, S, 1)` → `(B, S, H)`, making the expansion unnecessary.
- Replace `inputs_embeds[special_image_mask].numel() == image_features.numel()` validation with equivalent arithmetic `n_tokens * inputs_embeds.shape[-1] == image_features.numel()`, which avoids data-dependent boolean indexing and is more `torch.compile`-friendly.
- 71 files changed across llava, qwen2_vl, paligemma, gemma3n, chameleon, video_llava, idefics2/3, instructblip, blip_2, and many more.

## How this was developed

The core fix was first implemented and verified on `llava/modeling_llava.py`, then expanded to all other models following the same pattern using [Claude Code](https://claude.com/claude-code). Each file was reviewed to ensure the transformation was appropriate — files with genuinely different `expand_as` usage (e.g., `pe_audio` where the mask is later `.reshape()`-ed) were left unchanged.

## Test plan

- [x] Correctness verified: `masked_scatter` with broadcast `(B,S,1)` mask produces identical results to expanded `(B,S,H)` mask
- [x] `pytest tests/models/llava/test_modeling_llava.py -x -v -k "not slow"` — 136 passed
- [x] `pytest tests/models/qwen2_vl/test_modeling_qwen2_vl.py -x -v -k "not slow"` — 137 passed
- [x] `pytest tests/models/paligemma/test_modeling_paligemma.py -x -v -k "not slow"` — 124 passed
- [x] `ruff check` — all checks passed
- [x] `check_modular_conversion.py --fix_and_overwrite` — all generated files consistent
- [x] No duplicate PRs found (`gh pr list --search "expand_as placeholder mask"` / `"get_placeholder_mask"`)

This PR uses AI assistance (Claude Code). I have reviewed all changes and validated the behavior end-to-end.

Comments:
cc @zucchini-nlp it's an agent PR but looks high-quality to me, mostly because I'm a sucker for ways to avoid materializing broadcasts and tensor expansions :sweat_smile: 

Yea, this should be mostly copy-paste. I'm coming to this later today

Hi @syncdoth @zucchini-nlp, is this PR still on track to merge? It's currently mergeable_state: dirty and last activity was 2026-03-23. We've hit a real backward() crash on PaliGemma that this PR's .expand_as removal fixes, so we'd like to consume it. Happy to help with the rebase if useful.

Oops, sorry, lemme review it today 

This has been stale for a while, but the changes are pretty straightforward. If the high-levels are reviewed, I can resolve the merge conflicts and potentially expand this fix to newer models not currently included in this PR.

@aws-zhanxun just curious, was the `.backward` crash due to OOM? Fundamentally, all this PR does is removing unnecessary `expand_as`, which may sometimes help with OOM, but I didn't expect it to fix any other issue. If it did, that might be an unintended behavior.

Hi @syncdoth, thanks! Yes, it was an OOM during bwd, and removing the expand_as fixed it. No other behavior changes observed on our side. 

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_44907). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, colqwen2, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, fast_vlm, florence2, fuyu, gemma3, gemma3n, glm46v, llava, llava_next_video, qwen2_vl, qwen3_vl, paddleocr_vl, t5gemma2

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27338209440)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aria", "models/aya_vision", "models/blip_2", "models/chameleon", "models/cohere2_vision", "models/colqwen2", "models/deepseek_vl", "models/deepseek_vl_hybrid", "models/emu3", "models/ernie4_5_vl_moe", "models/fast_vlm", "models/florence2", "models/fuyu", "models/gemma3", "models/gemma3n", "models/glm46v", "models/llava", "models/llava_next_video", "models/paddleocr_vl", "models/qwen2_vl", "models/qwen3_vl", "models/t5gemma2"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27338209440)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [65628012](https://github.com/huggingface/transformers/commit/656280125edf8bca973f55272fbacd3f3069ef1a) | workflow commit (merge commit) |
| PR | [930efcc5](https://github.com/syncdoth/transformers/commit/930efcc5d04752a7fc61e81a7ecce750f22a99b7) | branch commit (from PR) |
| main | [dc9179e1](https://github.com/huggingface/transformers/commit/dc9179e1e92b85176cc1228b69ddc21c27b757ea) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


@bot /repo

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aria, aya_vision, blip_2, chameleon, cohere2_vision, colqwen2, cosmos3_omni, deepseek_ocr2, deepseek_vl, deepseek_vl_hybrid, emu3, ernie4_5_vl_moe, exaone4_5, fast_vlm, florence2, fuyu

@bot /style

Style fix fix runs successfully without any file modified.

run-slow: granite4_vision, deepseek_ocr, colqwen2, exaone4_5, qianfan_ocr

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27344703152)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/colqwen2", "models/exaone4_5", "models/granite4_vision", "models/qianfan_ocr"]
quantizations: []

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=44907&sha=cbc0d8

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/27344703152)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [cfe3c743](https://github.com/huggingface/transformers/commit/cfe3c7438befbab157896dd63c6567ca8de8005c) | workflow commit (merge commit) |
| PR | [cbc0d822](https://github.com/syncdoth/transformers/commit/cbc0d82202ea1447a024793279252fc9ff97ab0e) | branch commit (from PR) |
| main | [6b862157](https://github.com/huggingface/transformers/commit/6b8621578d28e2585382f418e701c8cc289fcc04) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


haha thanks for getting the work done. BTW, should we have worked on the `modular_*.py` instead of `modeling_*.py`? 

Yeah, the `modular` is the default and running `make fix-repo` will auto-copy its content when relevant to all models