audio pipeline support for initial_prompt?
Author: silvacarl2
Pull request: no
Labels: name: Feature request

### System Info

Is there a parameter someplace for audio pipeline support for initial_prompt?

like this: https://github.com/openai/whisper/discussions/963

$ whisper --help
optional arguments:
--initial_prompt INITIAL_PROMPT
optional text to provide as a prompt for the first window. (default: None)

$ whisper-ctranslate2 --help
optional arguments:
--initial_prompt INITIAL_PROMPT
optional text to provide as a prompt for the first window. (default: None)

### Who can help?

_No response_

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

audio pipeline-initial_paremeter

### Expected behavior

audio pipeline-initial_paremeter

Comments:
cc @sanchit-gandhi 

let me know if i am mistaken, but we cannot find initial_prompt in audio pipeline.

Hey @silvacarl2! The `pipeline` is a high-level helper function that is compatible with **all** audio models in the Transformers library. It's not meant to be fully comprehensive, but rather an easy API for all audio models over the common set of features (e.g. audio inputs, text outputs, batching, etc).

On the other hand, the `initial_prompt` is Whisper-specific argument. For example, there's no concept of `initial_prompt` for other speech models, like Wav2Vec2. Hence, it's not currently supported in the pipeline.

Given the prominence of Whisper as the current most popular model for ASR, I would be in favour of making an exception, and allowing the `initial_prompt` as a valid arg for the `pipeline`. WDYT here @ArthurZucker @Narsil @ylacombe?

you are a genius!  if that is possible, it would be great!!!!!!!!!!!!!!

just to be clear, whisper is the coolest thing we have seen in a long time, and i am sure everyone else thinks the same.

Well the pipeline is already pretty bloated with whisper stuff so no problem for me either. It's already supported (edit because what we don't support is the conditioning I think?)

I agree with you all, whisper is special enough. And I also agree it should be seen as an exception.

Any news for this? How can we force this initial_prompt first while waiting for the new release?

You can use the model outside of the pipeline for now 

Does use the model outside the pipeline also use a transformer? Sorry I am beginner in it. 

Hi, I have this code that as you suggest to use the model outside of the pipeline. But why in only transcribe the first portion of the audio. It seems only transcribe first 3 seconds of the audio but the audio is almost 30 minutes long. What am I doing wrong? Thank you

```python

from transformers import WhisperProcessor, WhisperForConditionalGeneration
import librosa
import torch
device = "cuda" if torch.cuda.is_available() else "cpu"


# Load the Whisper processor and model
processor = WhisperProcessor.from_pretrained("openai/whisper-tiny")
model = WhisperForConditionalGeneration.from_pretrained("openai/whisper-tiny").to(device)

# Load the audio file using librosa
file_path = "路人y2mate.com - 大陸年輕人薪資街頭調查 聽到這個數字新疆正妹直呼荒謬扯淡不可能中天朋友圈 CtiNews userjf1ro3lv8o_144p.wav"
audio_data, _ = librosa.load(file_path, sr=16000)  # Adjust the sample rate if needed

# Process the audio and generate output
input_features = processor.feature_extractor(audio_data, return_tensors="pt").input_features.to(device)
print(input_features.shape)
# Without prompt
# output_without_prompt = model.generate(input_features)
# print(processor.decode(output_without_prompt[0]))

# With prompt
prompt_ids = processor.get_prompt_ids("你好，這是一個中文範例")

# generate token ids by running model forward sequentially
predicted_ids = model.generate(input_features,max_length=100, prompt_ids=prompt_ids,language="Chinese")

# post-process token ids to text
transcription = processor.tokenizer.batch_decode(predicted_ids, skip_special_tokens=True)
print(transcription)](url)
```

That is expected, you need to iterate over the audio because the whipser Processor (and model) only accepts 30sec long inputs. Inviting you to read the documentations and tutorials related to this to get the hang of it! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@sanchit-gandhi @Narsil , Hi i have done the changes for this. And raised a PR (https://github.com/huggingface/transformers/pull/28556). Please let me know, if anything else is needed. Here is an example to use it. 

``` python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-small"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=15,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=device,
    processor=processor
)

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

# including timestamp
print(pipe(audio, initial_prompt = "Biswajit, Whisper", return_timestamps=True))

# without timestamp
print(pipe(audio, initial_prompt = "Biswajit, Whisper"))
```

THIS IS AWESOME!!!!!!!!!!!!!!!!!!  WILL CHECK IT OUT!!!!!!!!!!!!!!!!!!

will this also work with large-v2 and large-v3?

@silvacarl2  yes i have added that implementation to pipeline. yet PR needs approval. For the time being you can use my repository to try it out. `pip install git+https://github.com/Biswajit2902/transformers.git` for installation.

yes it will work for any whisper model.

Gentle ping @sanchit-gandhi (for here and the PR)

> @sanchit-gandhi @Narsil , Hi i have done the changes for this. And raised a PR (#28556). Please let me know, if anything else is needed. Here is an example to use it.
> 
> ```python
> from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
> from datasets import load_dataset
> 
> 
> device = "cuda:0" if torch.cuda.is_available() else "cpu"
> torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32
> 
> model_id = "openai/whisper-small"
> model = AutoModelForSpeechSeq2Seq.from_pretrained(
>     model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
> )
> model.to(device)
> 
> processor = AutoProcessor.from_pretrained(model_id)
> 
> pipe = pipeline(
>     "automatic-speech-recognition",
>     model=model,
>     tokenizer=processor.tokenizer,
>     feature_extractor=processor.feature_extractor,
>     max_new_tokens=128,
>     chunk_length_s=15,
>     batch_size=16,
>     torch_dtype=torch_dtype,
>     device=device,
>     processor=processor
> )
> 
> dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
> sample = dataset[0]["audio"]
> 
> # including timestamp
> print(pipe(audio, initial_prompt = "Biswajit, Whisper", return_timestamps=True))
> 
> # without timestamp
> print(pipe(audio, initial_prompt = "Biswajit, Whisper"))
> ```




i got this error:

# Fuse __init__ params and __call__ params without modifying the __init__ ones.

TypeError: AutomaticSpeechRecognitionPipeline._sanitize_parameters() got an unexpected keyword argument 'initial_prompt'

@Mutturaj-BEL , I have been working on this. Updated code in my own branch, Yet there are some issue with generation. You may refer for updates here #28556 .

will this be supported in the official release?

once the PR is merged, for sure!

When can we expect this PR to be merged ? Adding initial_prompt would solve a lot of issues on my end with traditional Chinese transcriptions.

Not sure, @Biswajit2902 has a few comments not adressed yet, I can review afterwards

Hey team & Biswajit. Really grateful for all the work. Just wanted to check in if this is a feature you're still implementing as it has been a while since last update

All the best

When is this going to be supported in the official release?

Hi team, any update on this? 

Any updates on this? Really needed for any serious development on Whisper. Thank you!

as far as i know it does this now.  we use faster-whisper library now though.

Hi, is there any update?