The `get_(text|image|audio|video)_features` methods have inconsistent output formats, needs aligning for Sentence Transformers
Author: tomaarsen
Pull request: no

Hello!

## Bug Report/Feature Request overview
* The `get_text_features`, `get_image_features`, etc. methods of many base models have inconsistent output formats.

## Details
The multimodal architectures in transformers can process all supported modalities simultaneously through the `forward` method of the base (i.e. without the head) model (e.g. `CLIPModel`), and many of these architectures also support getting embeddings of a single modality with methods in the form of `get_(text|image|audio|video)_features`, e.g. [`get_text_features`](https://github.com/huggingface/transformers/blob/b6055550a15a8fab367cf983b743ff68cc58d81a/src/transformers/models/clip/modeling_clip.py#L784-L819) and [`get_image_features`](https://github.com/huggingface/transformers/blob/b6055550a15a8fab367cf983b743ff68cc58d81a/src/transformers/models/clip/modeling_clip.py#L821-L858) for CLIP. 

Without these methods, it's not possible to get embeddings/hidden states of a single modality without requiring the other modalities as well, e.g. getting text embeddings for CLIP without passing an image as well.

For context, we have:
* 20 architectures with `get_text_features`
* 64 architectures with `get_image_features`
* 7 architectures with `get_audio_features`
* 15 architectures with `get_video_features`

These methods generally output pooled embeddings, i.e. the modality-specific dimension (`seq_length`, `width`, `height`) are pooled away, leaving only a shape of `(batch_size, num_dimensions)`, but this is not consistently applied in `transformers`. 

For example, in the 21 architectures that implement `get_text_features`, `blip_2` its [`get_text_features`](https://github.com/huggingface/transformers/blob/47e795b874b5393802d5bb6ab17bda45183ef711/src/transformers/models/blip_2/modeling_blip_2.py#L1067-L1129) outputs the `logits` of the language model, e.g. with a shape `(batch_size, seq_length, vocab_size)`.

See here a script to quickly reproduce the discrepancy/misalignment:

```python
import torch
from transformers import AutoTokenizer, AutoModel

# model_name = "Salesforce/blip2-opt-2.7b"
model_name = "openai/clip-vit-base-patch32"
model = AutoModel.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)
print(type(model))

inputs = tokenizer(["a photo of a cat"], padding=True, return_tensors="pt")
with torch.inference_mode():
    text_features = model.get_text_features(**inputs)
print(text_features.shape)
# torch.Size([1, 6, 50304])
# torch.Size([1, 512])
```

I see the same with audio, where [`get_image_features`](https://github.com/huggingface/transformers/blob/b6055550a15a8fab367cf983b743ff68cc58d81a/src/transformers/models/deepseek_vl/modeling_deepseek_vl.py#L156-L159) from `deepseek_vl` returns `(batch_size, seq_length, num_dimensions)` with `torch.Size([1, 576, 2048])`, while the exact same script with `clip` its [`get_image_features`](https://github.com/huggingface/transformers/blob/47e795b874b5393802d5bb6ab17bda45183ef711/src/transformers/models/clip/modeling_clip.py#L821-L858) returns `(batch_size, num_dimensions)` with `torch.Size([1, 512])`

See here a script to quickly reproduce the discrepancy/misalignment, but now for images of multi-modal models:

```python
import requests
from PIL import Image
from transformers import AutoImageProcessor, AutoModel
import torch

url = "http://images.cocodataset.org/val2017/000000039769.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# model_name = "deepseek-community/deepseek-vl-1.3b-base"
model_name = "openai/clip-vit-base-patch32"
processor = AutoImageProcessor.from_pretrained(model_name)
model = AutoModel.from_pretrained(model_name)
print(type(model))

inputs = processor(images=image, return_tensors="pt")
with torch.inference_mode():
    image_features = model.get_image_features(**inputs)
print(image_features.shape)
# torch.Size([1, 576, 2048])
# torch.Size([1, 512])
```

I didn't look into the audio and video ones, but even short glances show that there's large discrepancies there too. The `get_video_features` from qwel3_vl even returns a tuple of `image_embeds` and `deepstack_image_embeds`, and for audio I see pooled embeddings (clap, audioflamingo3) and unpooled embeddings (qwen2_5_omni), and tuples (gemma3n).

I think it's crucial that we align the output structures of these methods. Beyond that, I think it's very valuable if we expose the non-pooled embeddings (e.g. `last_hidden_states`) via these methods as well. This would allow for much stronger multi-modality support in Sentence Transformers, for example. 

I propose that we introduce a new parameter to each of the `get_(text|image|audio|video)_features` methods, something along the lines of `pooling: bool = True`. This keeps the behaviour identical for most models, but aligns the expected output for all models (of a modality), while also allowing Sentence Transformers and related projects to access the modality-specific embeddings without an arbitrary pooling procedure being applied on top. This accessibility is very important for third party libraries to properly use multi-modal models for new/novel use cases.

cc @ArthurZucker @Cyrilvallez @zucchini-nlp 
cc @LysandreJik 

- Tom Aarsen

Comments:
I think we have two large groups of models with different behavior and expectation from `get_xxx_features` method. CLIP-like models would return a pooled embedding (if there is a pooling layer, otherwise last hidden state) so that it can be used by users for image-text matching or simply to encode an image

Vision LLMs however return an image encoding ready to be concatenated with text embeddings and each model has their own way of post-processing these encodings. Some models do pooling, some do pixel shuffle and some even return several outputs (qwen3-vl). We usually use this utility 1) in the model's forward so the code is more modularized and can be easily overriden new models are dded and 2) we use it in vLLM integration which expects to get/cache image embeddings right before the `concatenation` operation. 

I agree that we could standardize it between the two groups of models, especially if we want to make it easier for Sentence Transformers to use VLMs as encoding models. Personally, I prefer to keep the VLM-style and we could return more options when requested in a NamedTuple format. WDYT?

> I agree that we could standardize it between the two groups of models, especially if we want to make it easier for Sentence Transformers to use VLMs as encoding models. Personally, I prefer to keep the VLM-style and we could return more options when requested in a NamedTuple format. WDYT?

With the VLM-style, do you refer to "pooled embeddings" as the default option? I.e. always a `(batch_size, hidden_size)` format?
And the NamedTuple, should that only be returned if some parameter is specified? I was thinking that perhaps we could create more subclasses of the `ModelOutput` for this instead (i.e. OrderedDict-style instead of NamedTuple-style).

- Tom Aarsen

Yeah, I mean "pooled" outputs as default single tensor instead of returning a Tuple/Dict. Otherwise it will be quite breaking and afaik the helper is used quite commonly. So, I prefer to leave as is by default and `return_dict` when asked similar to how model's forward works

No big issue for me, whether it's a ModelOutput or NamedTuple. We just need to get a set of common output keys that will work for all models :)

I think returning a `(batch_size, hidden_size)` format by default, with a `return_dict` that returns a `ModelOutput` subclass would be perfect, with `pooler_output` and `last_hidden_state` as the common output keys, and more options for specific architectures that require it.

- Tom Aarsen

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Still being worked on.

- Tom Aarsen