Prompt_ids vs. decoder_input_ids in Whisper
Author: vymao
Pull request: no

### Feature request

I am trying to understand the different between adding prior text to `prompt_ids` vs. `decoder_input_ids` when [generating text via Whisper](https://huggingface.co/docs/transformers/v4.36.1/en/model_doc/whisper#transformers.WhisperForConditionalGeneration). The documentation is not very clear on how these differ implementation-wise; AFAIK, it seems like using `prompt_ids` will lead to `forced_input_ids` being modified [here](https://github.com/huggingface/transformers/blob/v4.36.1/src/transformers/models/whisper/modeling_whisper.py#L2211-L2215). But I'm not sure how exactly using `decoder_input_ids` differs from this.

### Motivation

To add context to the whisper transcription. For example, if the model previously transcribed `I have a` in a streaming fashion, I would like to add this as "context" into the model to help it predict the next word. I believe the actual OpenAI Whisper implementation has a feature called "prefix" that does this.

### Your contribution

Will try.

Comments:
Hey @vymao, `prompt_ids` basically refers to the input tokens or token IDs provided to the model before generating text and it serves as an initial context for the model to begin generating text.

On the other hand, `decoder_input_ids` are mainly used in sequence-to-sequence models or in models with a decoder part. For example, Transformer architectures with encoder-decoder structure. So `decoder_input_ids` are inputs provided to the decoder part of a sequence-to-sequence model and they help guide the generation of subsequent tokens in the sequence.

When it comes to generating text via `Whisper`, both `prompt_ids` and `decoder_input_ids` can be used to provide context to guide the model's text generation. Also, the `prefix` feature in `Whisper` mainly uses either `prompt_ids` or `decoder_input_ids` or a combination of both to provide context to the model.

The implementation difference between `prompt_ids` and `decoder_input_ids` is that `prompt_ids` usually provide the initial context while `decoder_input_ids` guides the decoding or generation process, mostly when it involves encoder-decoder architectures.

@254guru


Thanks. I'm still slightly confused: when you say `prompt_ids` are used to provide initial context, isn't that still on the decoder side before the actual generated text? How is this different from using `decoder_input_ids`? 

Maybe for @sanchit-gandhi or @ylacombe 

Hey @vymao, I'm not a Whisper expert yet but as I understand and as the [documentation](https://huggingface.co/docs/transformers/v4.36.1/en/model_doc/whisper#transformers.WhisperForConditionalGeneration.generate.prompt_ids) suggests, `prompt_ids` are created by using the tokenizer's or the processor's `get_prompt_ids`.

https://github.com/huggingface/transformers/blob/3cefac1d974db5e2825a0cb2b842883a628be7a0/src/transformers/models/whisper/tokenization_whisper.py#L827-L836

As you can see from the code, get_prompt_ids handles the input text so you don't have to worry about special tokens that need to be inserted to tell the model that this text is context and not the start of the transcription.

Then the code processes the `prompt_ids` in place of the `decoder_input_ids`.

In other words, you can use  `prompt_ids` obtained from `get_prompt_ids` if you want to pass a context to Whisper.  `decoder_input_ids` is much more flexible: you could reproduce `prompt_ids` obtained from `get_prompt_ids` or use it to have a more advanced use of Whisper.

I hope that it helps!

cc @sanchit-gandhi or @ArthurZucker if you want to correct me or give a more advanced explanation



Hey @vymao,

That's a very good question! In a nutshell, `decoder_input_ids` and `prompt_ids` are the same thing. The allow you to prompt Whisper on a specific prefix just like it's explained here: https://platform.openai.com/docs/guides/speech-to-text/prompting

Please use `prompt_ids` for the moment and don't use `decoder_input_ids`. I'm working on improving the docs and usability of Whisper at the moment which this PR: https://github.com/huggingface/transformers/pull/27658

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.