can't translate audio to text when use seamless-m4t-v2-large
Author: ningpp
Pull request: no
Labels: name: bug

### System Info

transformers==4.42.4   python 3.11  windows

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
model = AutoModelForSpeechSeq2Seq.from_pretrained(
                get_audio_model_path(),
                torch_dtype=torch_dtype,
                low_cpu_mem_usage=False,
                use_safetensors=True
            )
model.to(get_audio_device())

processor = AutoProcessor.from_pretrained(get_audio_model_path())


pipe = pipeline(
    task="automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=2048,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=get_audio_device(),
)

audio_result = pipe(dest_audio_file, generate_kwargs={"tgt_lang": "cmn"})
```


error log
```
> File "D:\code\chat-b03\libs\chatchat-server\chatchat\zhilin\server\audio\speech_to_text.py", line 64, in speech_text_translation_file
    audio_result = pipe(dest_audio_file, generate_kwargs={"tgt_lang": "cmn"})
                   │    └ 'C:\\Users\\ningpp\\AppData\\Local\\Temp\\a71dd2b661b143f6bfd84224eaa4ee72.mp3'
                   └ <transformers.pipelines.automatic_speech_recognition.AutomaticSpeechRecognitionPipeline object at 0x000001B9324633D0>

  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\automatic_speech_recognition.py", line 284, in __call__
    return super().__call__(inputs, **kwargs)
                            │         └ {'generate_kwargs': {'tgt_lang': 'cmn'}}
                            └ 'C:\\Users\\ningpp\\AppData\\Local\\Temp\\a71dd2b661b143f6bfd84224eaa4ee72.mp3'
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\base.py", line 1246, in __call__
    return next(
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\pt_utils.py", line 124, in __next__
    item = next(self.iterator)
                │    └ <transformers.pipelines.pt_utils.PipelinePackIterator object at 0x000001B938AB4290>
                └ <transformers.pipelines.pt_utils.PipelineIterator object at 0x000001B938AB4490>
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\pt_utils.py", line 269, in __next__
    processed = self.infer(next(self.iterator), **self.params)
                │    │          │    │            │    └ {'max_new_tokens': 2048, 'tgt_lang': 'cmn'}
                │    │          │    │            └ <transformers.pipelines.pt_utils.PipelinePackIterator object at 0x000001B938AB4290>
                │    │          │    └ <torch.utils.data.dataloader._SingleProcessDataLoaderIter object at 0x000001B938AB44D0>
                │    │          └ <transformers.pipelines.pt_utils.PipelinePackIterator object at 0x000001B938AB4290>
                │    └ <bound method Pipeline.forward of <transformers.pipelines.automatic_speech_recognition.AutomaticSpeechRecognitionPipeline obj...
                └ <transformers.pipelines.pt_utils.PipelinePackIterator object at 0x000001B938AB4290>
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\base.py", line 1161, in forward
    model_outputs = self._forward(model_inputs, **forward_params)
                    │    │        │               └ {'max_new_tokens': 2048, 'tgt_lang': 'cmn'}
                    │    │        └ {}
                    │    └ <function AutomaticSpeechRecognitionPipeline._forward at 0x000001B97CEE6480>
                    └ <transformers.pipelines.automatic_speech_recognition.AutomaticSpeechRecognitionPipeline object at 0x000001B9324633D0>
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\pipelines\automatic_speech_recognition.py", line 504, in _forward
    tokens = self.model.generate(
             │    │     └ <function SeamlessM4Tv2ForSpeechToText.generate at 0x000001B93292FBA0>
             │    └ SeamlessM4Tv2ForSpeechToText(
             │        (shared): Embedding(256102, 1024, padding_idx=0)
             │        (speech_encoder): SeamlessM4Tv2SpeechEncod...
             └ <transformers.pipelines.automatic_speech_recognition.AutomaticSpeechRecognitionPipeline object at 0x000001B9324633D0>
  File "D:\code\chat-b03\libs\chatchat-server\.venv\Lib\site-packages\transformers\models\seamless_m4t_v2\modeling_seamless_m4t_v2.py", line 3430, in generate
    batch_size = len(inputs)
                     └ None

TypeError: object of type 'NoneType' has no len()
```

### Expected behavior


https://github.com/ningpp/transformers/commit/603f35898e0f067d12f3079fc0199830fc08c810

Comments:
cc @ylacombe 

Hey @ningpp, thanks for opening this issue!

Do you think you could send a full snippet to reproduce the issue? with the current snippet, we don't have the `get_audio_model_path()` and a dummy `dest_audio_file` so no way for me to reproduce.

Also, have you been able to run the model without the pipeline? If so, it'll help us pinpoint where exactly the issue occurs.

Thanks

@ylacombe  unzip this file to get a sample audio file
[a71dd2b661b143f6bfd84224eaa4ee71.zip](https://github.com/user-attachments/files/16409935/a71dd2b661b143f6bfd84224eaa4ee71.zip)

```

import torch
from transformers import AutoProcessor, AutoModelForSpeechSeq2Seq, pipeline

torch_dtype = torch.float32


def get_audio_device() -> str:
    return "cpu"


def get_audio_model_path() -> str:
    return "/models/facebook/seamless-m4t-v2-large"


dest_audio_file = "/tmp/a71dd2b661b143f6bfd84224eaa4ee71.mp3"


model = AutoModelForSpeechSeq2Seq.from_pretrained(
                get_audio_model_path(),
                torch_dtype=torch_dtype,
                low_cpu_mem_usage=False,
                use_safetensors=True
            )
model.to(get_audio_device())

processor = AutoProcessor.from_pretrained(get_audio_model_path())


pipe = pipeline(
    task="automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=2048,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=get_audio_device(),
)

audio_result = pipe(dest_audio_file, generate_kwargs={"tgt_lang": "cmn"})
```

Hey @ningpp, https://github.com/huggingface/transformers/pull/32296 should have fixed the issue! Let me know if you have questions

@ylacombe LGTM! thank you