Add initial_prompt
Author: tinderlord
Pull request: no

### Feature request

This parameter will allow a string variable to be passed when encoding

### Motivation

To allow better transcription based on prompt and audio

### Your contribution

Not at the moment

Comments:
Hi @tinderlord - thanks for opening an issue! 

Could you provide a bit more information about the feature - ideally with an example code snippet of how it would be used and details about the models this would apply to (I'm assuming whisper?)

cc @sanchit-gandhi 

Note that this is available in Transformers after the PR #22496:
```python
from transformers import WhisperProcessor, WhisperForConditionalGeneration
from datasets import load_dataset

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
input_speech = dataset[3]["audio"]["array"]

processor = WhisperProcessor.from_pretrained("distil-whisper/distil-large-v2")
model = WhisperForConditionalGeneration.from_pretrained("distil-whisper/distil-large-v2")
input_features = processor(input_speech, return_tensors="pt").input_features

# --- Without prompt ---
output_without_prompt = model.generate(input_features)
print(processor.decode(output_without_prompt[0]))
# 



 He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.


# --- With prompt ---
# Let's change the spelling of "Leighton" -> "Layton" by passing it as a prompt
prompt_ids = processor.get_prompt_ids("Layton")
output_with_prompt = model.generate(input_features, prompt_ids=prompt_ids)
print(processor.decode(output_with_prompt[0]))
# 
 Layton



 He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.

```

Is this what you're looking for @tinderlord?

@sanchit-gandhi, can you tell how to use this prompt feature witn whisper asr pipeline?

It is not supported yet but planned

@ArthurZucker, thanks for your answer. Maybe you can help me with using initial prompt with batching asr without pipeline?

> Note that this is available in Transformers after the PR #22496:
> 
> ```python
> from transformers import WhisperProcessor, WhisperForConditionalGeneration
> from datasets import load_dataset
> 
> dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
> input_speech = dataset[3]["audio"]["array"]
> 
> processor = WhisperProcessor.from_pretrained("distil-whisper/distil-large-v2")
> model = WhisperForConditionalGeneration.from_pretrained("distil-whisper/distil-large-v2")
> input_features = processor(input_speech, return_tensors="pt").input_features
> 
> # --- Without prompt ---
> output_without_prompt = model.generate(input_features)
> print(processor.decode(output_without_prompt[0]))
> # 



 He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.

> 
> # --- With prompt ---
> # Let's change the spelling of "Leighton" -> "Layton" by passing it as a prompt
> prompt_ids = processor.get_prompt_ids("Layton")
> output_with_prompt = model.generate(input_features, prompt_ids=prompt_ids)
> print(processor.decode(output_with_prompt[0]))
> # 
 Layton



 He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.

> ```
> 
> Is this what you're looking for @tinderlord?

Pretty sure @sanchit-gandhi's answer is what you are looking for! 🤗 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Yes, many thanks. This feature is what i was looking for. 

You can also use it with the pipeline class as follows:

```python
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "distil-whisper/distil-large-v2"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    torch_dtype=torch_dtype,
    device=device,
)

dataset = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = dataset[3]["audio"]

# --- Without prompt ---
result = pipe(sample.copy())
print("Without prompt:", result["text"])
# He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.

# --- With prompt ---
# Let's change the spelling of "Leighton" -> "Layton" by passing it as a prompt
prompt = "Layton"
prompt_ids = processor.get_prompt_ids(prompt, return_tensors="pt").to(device)
result = pipe(sample, generate_kwargs={"prompt_ids": prompt_ids})
print("With prompt:", result["text"][len(prompt) + 1:])
# He has grave doubts whether Sir Frederick Layton's work is really Greek after all, and can discover in it but little of Rocky Ithaca.
```