Repository: huggingface/transformers
Pull request: Standardize audio/vision embedding function name for multimodal models

# What does this PR do?
Make all multimodal models share the same function name for audio and vision. This function should encapsulate all of the encoder module code up to the fusion with the prompt embeddings when doing early fusion.

This makes it so that we can rely on this function name downstream:
- Audio: https://github.com/huggingface/optimum-executorch/blob/main/optimum/exporters/executorch/integrations.py#L132 (currently it is `get_audio_embeds` but we will change it to `get_audio_features` after this PR is landed.
- Vision: https://github.com/huggingface/optimum-executorch/blob/main/optimum/exporters/executorch/integrations.py#L78

All audio models now follow the `get_audio_features` API and early fusion pattern:
- [Granite Speech](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/granite_speech/modeling_granite_speech.py#L349)
- Voxtral (changed in this PR)
- [Qwen2.5 Omni](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_5_omni/modeling_qwen2_5_omni.py#L1720)
- [Gemma 3n](https://github.com/huggingface/transformers/blob/main/src/transformers/models/gemma3n/modular_gemma3n.py#L2457)

Most vision models already follow the `get_image_features` API and early fusion pattern (vision models taken from [this](https://github.com/huggingface/transformers/blob/main/src/transformers/models/auto/modeling_auto.py#L997) list). The few that don't are either unrelated / cross attention-based:
- [Aria](https://github.com/huggingface/transformers/blob/main/src/transformers/models/aria/modular_aria.py#L1356)
- [Aya](https://github.com/huggingface/transformers/blob/main/src/transformers/models/aya_vision/modular_aya_vision.py#L110)
- Blip - N/A, not image-text-to-text
- [Blip2](https://github.com/huggingface/transformers/blob/main/src/transformers/models/blip_2/modeling_blip_2.py#L1739)
- [Chameleon](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/chameleon/modeling_chameleon.py#L883)
- [Cohere2 Vision](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/cohere2_vision/modular_cohere2_vision.py#L164)
- [Deepseek VL](https://github.com/huggingface/transformers/blob/main/src/transformers/models/janus/modular_janus.py#L904C9-L904C27) (derived from Janus)
- Deepseek VL Hybrid - same as Deepseek VL
- [Emu3](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/emu3/modular_emu3.py#L933)
- Evolla - N/A, for proteins (?)
- [Florence2](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/florence2/modular_florence2.py#L1536) (this does image features -> seq2seq instead of decoder)
- [Fuyu](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/fuyu/modeling_fuyu.py#L136)
- [Gemma3](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/gemma3/modular_gemma3.py#L763) - already enabled on Optimum ET
- [Gemma3n](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/gemma3n/modular_gemma3n.py#L2250)
- Git - not image-text-to-text
- [GLM 4.1V](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/glm4v/modeling_glm4v.py#L1121)
- GLM 4.1V MOE - same as GLM 4.1V
- [GOT-OCR2](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/got_ocr2/modular_got_ocr2.py#L308)
- ❌ IDEFICS - cross attention
- [IDEFICS 2](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/idefics2/modeling_idefics2.py#L860)
- [IDEFICS 3](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/idefics3/modeling_idefics3.py#L610)
- [InstructBLIP](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/instructblip/modeling_instructblip.py#L1263)
- [InternVL](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/internvl/modular_internvl.py#L493)
- ❌ KOSMOS-2 - this one has [`get_image_features`](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/kosmos2/modeling_kosmos2.py#L1518) but after that it's structured a bit differently, needing to pass in a `image_embeds_position_mask`, will need some work to export
- ❌ KOSMOS-2.5 - same as above
- [Llama4](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/llama4/modeling_llama4.py#L1173)
- [LLaVA](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/llava/modeling_llava.py#L154)
- [LLaVA-NeXT](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/llava_next/modeling_llava_next.py#L349)
- [LLaVA-NeXT-Video](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/llava_next_video/modeling_llava_next_video.py#L401)
- [LLaVA-OneVision](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/llava_next_video/modeling_llava_next_video.py#L401)
- [Mistral 3](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/mistral3/modular_mistral3.py#L121)
- ❌ Mllama (Llama 3.2-Vision) - cross attention
- [Ovis2](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/ovis2/modular_ovis2.py#L228)
- [PaliGemma](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/paligemma/modeling_paligemma.py#L232)
- [PerceptionLM](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/perception_lm/modular_perception_lm.py#L149)
- Pix2Struct - not image-text-to-text
- Pixtral - same as LLaVA
- [Qwen2.5-VL](https://github.com/huggingface/transformers/blob/main/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1093)
- [Qwen2-VL](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/qwen2_vl/modeling_qwen2_vl.py#L1093)
- [Qwen3-VL](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/qwen3_vl/modeling_qwen3_vl.py#L1050)
- Qwen3-VL MOE - same as Qwen3-VL
- [SmolVLM](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/smolvlm/modular_smolvlm.py#L198)
- ❌ UDOP
- [ViP-LLaVA](https://github.com/huggingface/transformers/blob/6e50a8afb2540ac1acaa4b62cf1dd5f1170f6511/src/transformers/models/vipllava/modular_vipllava.py#L75)
- ❌ VisionEncoderDecoderModel - not sure this model is but it's cross attention-based

Following this PR, we would like to move [`MultimodalTextToTextExportableModule`](https://github.com/huggingface/optimum-executorch/blob/main/optimum/exporters/executorch/integrations.py#L136) from Optimum ET to [`transformers/integrations/executorch.py`](https://github.com/huggingface/transformers/blob/main/src/transformers/integrations/executorch.py) and add multimodal export tests marked with `@pytest.mark.torch_export_test` for each multimodal model.

cc @ArthurZucker @zucchini-nlp 

Changed files:
filename: src/transformers/models/voxtral/modeling_voxtral.py
patch: @@ -20,6 +20,7 @@
 # limitations under the License.
 
 import math
+import warnings
 from typing import Callable, Optional, Union
 
 import torch
@@ -431,7 +432,7 @@ def set_decoder(self, decoder):
     def get_decoder(self):
         return self.language_model.get_decoder()
 
-    def get_audio_embeds(self, input_features: torch.FloatTensor):
+    def get_audio_features(self, input_features: torch.FloatTensor):
         """
         This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.
         Args:
@@ -452,6 +453,12 @@ def get_audio_embeds(self, input_features: torch.FloatTensor):
         audio_embeds = self.multi_modal_projector(audio_hidden_states)
         return audio_embeds
 
+    def get_audio_embeds(self, input_features: torch.FloatTensor):
+        warnings.warn(
+            "The method `get_audio_embeds` is deprecated. Please use `get_audio_features` instead.", FutureWarning
+        )
+        return self.get_audio_features(input_features)
+
     @can_return_tuple
     @auto_docstring
     def forward(
@@ -505,7 +512,7 @@ def forward(
             inputs_embeds = self.get_input_embeddings()(input_ids)
 
         if input_features is not None and input_ids is not None:
-            audio_embeds = self.get_audio_embeds(input_features)
+            audio_embeds = self.get_audio_features(input_features)
 
             # replace text-audio token placeholders with audio embeddings
             audio_token_mask = (input_ids == self.config.audio_token_id).unsqueeze(-1)

filename: src/transformers/models/voxtral/modular_voxtral.py
patch: @@ -13,6 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+import warnings
 from typing import Optional, Union
 
 import torch
@@ -166,7 +167,7 @@ def set_decoder(self, decoder):
     def get_decoder(self):
         return self.language_model.get_decoder()
 
-    def get_audio_embeds(self, input_features: torch.FloatTensor):
+    def get_audio_features(self, input_features: torch.FloatTensor):
         """
         This method is used to get the audio embeddings from input features (a log mel spectrogram), meaning inferring the audio encoder and the multi-modal projector.
         Args:
@@ -187,6 +188,12 @@ def get_audio_embeds(self, input_features: torch.FloatTensor):
         audio_embeds = self.multi_modal_projector(audio_hidden_states)
         return audio_embeds
 
+    def get_audio_embeds(self, input_features: torch.FloatTensor):
+        warnings.warn(
+            "The method `get_audio_embeds` is deprecated. Please use `get_audio_features` instead.", FutureWarning
+        )
+        return self.get_audio_features(input_features)
+
     @can_return_tuple
     @auto_docstring
     def forward(
@@ -240,7 +247,7 @@ def forward(
             inputs_embeds = self.get_input_embeddings()(input_ids)
 
         if input_features is not None and input_ids is not None:
-            audio_embeds = self.get_audio_embeds(input_features)
+            audio_embeds = self.get_audio_features(input_features)
 
             # replace text-audio token placeholders with audio embeddings
             audio_token_mask = (input_ids == self.config.audio_token_id).unsqueeze(-1)