Consistent output shape from `get_image_features`
Author: zucchini-nlp
Pull request: yes

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=46405)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46405)
<!-- ci-dashboard-badge:end -->

# What does this PR do?

as per title, branches off from https://github.com/huggingface/transformers/pull/45783

The pooled image (OR video, NOT audio yet) feature output will now always satisfy three cond, where image features can be a list or a 3D tensor:
1) len(image_features) = len(input_images) # NOTE: or num_videos, not number of total video frames
2) image_features[0].ndim == 2
3) image_features[0].shape == {{actual seq length of this image, LM hidden size}}

Could make it complete BC and return the "correct" output when a certain flag is passed, but for now decided to update directly smaller utility fn in a breaking way

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_46405). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

run-slow: aya_vision, cohere2_vision, deepseek_ocr2, gemma4, paddleocr_vl, qwen2_5_omni, qwen3_omni_moe

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/26944813797)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/aya_vision", "models/cohere2_vision", "models/deepseek_ocr2", "models/gemma4", "models/paddleocr_vl", "models/qwen2_5_omni", "models/qwen3_omni_moe"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/26944813797)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [5e5f7bbe](https://github.com/huggingface/transformers/commit/5e5f7bbe8b100b1f3b38a25d1ef34a9e04b19030) | workflow commit (merge commit) |
| PR | [52788433](https://github.com/zucchini-nlp/transformers/commit/5278843334932a9eb83e2112d4192c5361d3f759) | branch commit (from PR) |
| main | [b07d99be](https://github.com/huggingface/transformers/commit/b07d99be866747683172d1729c979b2a64cba23a) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


We also need to add :rotating_light: for sure

> Do we not cat in get image features, why?
Answered in comments, to get clearly split encodings per image so we don't have to guess. It'll make to easy to get embeddings of image-i and expand/crop/do whatever you need 

> We sometimes have to calculate the expected number of tokens produced by the vision tower; would it make sense to align across processor/modeling by having some global function?

You mean gemma4 (which I also replied is based on modeling, not processor)? Otherwise we always pass the number of grids (as in qwen) or just split based on input pixel shape

Oh yeah, Tom requested this in the past for `last_hidden_states` which I personally didn't need for generation 😅  Leaving it open, and working only on `pooler_output` for now

Yea I think we are mostly fine then, still want to wait on @tomaarsen if he has any opinions on this and maybe @remi-or using it for CB(?)

yeah, let's wait for ppl affected. I also commented on CB to let Remi know :)

In general, anything that makes shapes more static or standardize inputs is good for CB, so this is great!  

The only problem that this could create is if `torch.split` is called with a number of splits that cannot be determined beforehand, which would not be CUDA graphable. Eventually, we will want CUDA graphs around the encoder.
But, since multimodal for CB has not landed yet, and the first iteration of that won't have CUDA graphs around the encoder, maybe we solve that problem when we get to it. Then, maybe an option to return the inputs before the split would do the trick.

I updated my ST test suite a bit [here](https://github.com/huggingface/sentence-transformers/pull/3829), and then ran it in conjunction with this branch: no issues. The code also looks solid from my perspective, i.e. more standardized is more better 😄 

@remi-or 's concern is fair though.

- Tom Aarsen

> if torch.split is called with a number of splits that cannot be determined beforehand

indeed, the number if splits would be dynamic each call depending on input image. Even though it is part of `inputs`, the split will cause graphbreak. And now that I think about it, we're interfering with Ilyas' efforts to make VLMs exportable 😿 

@vasqu I'll tag you after checking in with exporters and finding a better way to make everyone happy 

I think we can revisit this after #41992? I'm not sure but I don't think it would interfere with Ilyas work as we forward the inputs once and capture the necessary metadata based on that - so yea it should be fine on that side

@vasqu , let's merge? I rebased main and fixed new models

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aya_vision, cohere2_vision, deepseek_ocr2, diffusion_gemma, gemma4, inkling, kimi_k25, paddleocr_vl, qwen2_5_omni, qwen3_omni_moe

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=46405)
**Latest run:** [29578279005:1](https://github.com/huggingface/transformers/actions/runs/29578279005)
**Result:** `failure` | **Jobs:** 15 | **Tests:** 163,185 | **Failures:** 21 | **Duration:** 11h 36m

<!-- ci-dashboard-recap:end -->