Feature Update [added `initial_prompt` support for automatic-speech-recognition whisper pipeline]
Author: Biswajit2902
Pull request: yes
Labels: name: Core: Pipeline

name: Audio

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (feature)
- `initial_prompt` support for whisper Pipeline (automatic-speech-recognition) 

## Before submitting
- [x] Added initial_prompt as an option for whisper model
- [x] To handle initial prompt `processor` considered as optional parameter
- [x] Current implementation supports only Torch version of decoding.
- [x] how to use initial prompt; 

``` python
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset
import torch

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-small"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=15,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=device,
    processor=processor
)


dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]
audio = dataset[0]["audio"]["array"]
sampling_rate = dataset[0]["audio"]["sampling_rate"]

# including timestamp
print(pipe(audio, initial_prompt = "Biswajit, Whisper", return_timestamps=True))

# without timestamp
print(pipe(audio, initial_prompt = "Biswajit, Whisper"))
```

## Who can review?

Anyone in the community is free to review the PR once the tests have passed. @sanchit-gandhi , @Narsil, Can anyone help to take this PR forward please. Let me know, if anything is needed.

fixes #27317
<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker and @younesbelkada
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @gante
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @pacman100

Integrations:

- deepspeed: HF Trainer/Accelerate: @pacman100
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc and @younesbelkada

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi thank you your code saved my day! I think line 535 needs to modify a bit `prompt_tensor = torch.tensor(generate_kwargs["prompt_ids"], dtype=out["tokens"].dtype).cuda() if is_torch_cuda_available else torch.tensor(generate_kwargs["prompt_ids"], dtype=out["tokens"].dtype)`, and add `is_torch_cuda_available` to line 22. without cuda it'll run on cpu which is a lot slower. 

@kaminwong , this is just to modify the output sequence to avoid showing `inital_prompt` in transcription.

Actual generation has device handles in below line.
``` python
           tokens = self.model.generate(
                attention_mask=attention_mask,
                **generate_kwargs,
            )
```

Apart from this token decoding part is serialised implementation which has no effect, that can be misuse of GPU.

Thanks for the reply! But if I don't make that changes I get the following error, so I assume `prompt_tensor` needs to be in cuda if device is also in cuda? Or is there any other way to correct the error? Thank you for your time.

`File "/.../python3.10/site-packages/transformers/pipelines/automatic_speech_recognition.py", line 538, in _forward
    if (tmp_tokens[0:nprompt_token] == prompt_tensor).sum() == nprompt_token:
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!`

I followed the code you posted:
```from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline

device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-small"
model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=15,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=device,
    processor=processor
)



@kaminwong , Thank you for addressing. I understood the issue. let me verify and reolved it.

@kaminwong , you can pull latest commit and install it should work now. its fixed.

Thank you for the elegant solution. It works now!

Gentle ping @sanchit-gandhi for review 

@amyeroberts is there any plan to close this in near future? or will it take time?


@Biswajit2902 Once @sanchit-gandhi has reviewed and approved, the PR will need a final review from a maintainer. Once approved, then the PR can be merged in. 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

@Biswajit2902 any new updates? let me know if you need help

@thomasmol I will update on this soon. was busy since two weeks. Thank you for the reminder.

@thomasmol @sanchit-gandhi , i see below conflict in `AutomaticSpeechRecognitionPipeline._sanitize_parameters`;

```
<<<<<<< main
            forward_params["generate_kwargs"]["max_new_tokens"] = max_new_tokens
        if initial_prompt is not None:
            forward_params["generate_kwargs"]["initial_prompt"] = initial_prompt
=======
            forward_params["max_new_tokens"] = max_new_tokens
>>>>>>> main
```

I want to understand why we removed `generate_kwargs` from forward_params. Also initial_prompt.

My changes before were working fine. But after this, there seems have some bug. I am working on resolving it. So need your input on this.

Hey @Biswajit2902 - you can read the motivation for this change [here](https://github.com/huggingface/transformers/pull/29614/files#r1524786802). Essentially, we're unifying the `forward_params` and `generate_kwargs` in `_sanitize_parameters`. However, for the purposes of your feature, you should strive to put the `initial_prompt` under `preprocess_params`:
```python
preprocess_params["initial_prompt"] = initial_prompt
```
And then convert the text prompt to token ids in the `preprocess` method, which will then be passed to `_forward`.

@sanchit-gandhi , Thanks for the pointer. Sorry got super busy could go back review. Will do it soon and close it.

@sanchit-gandhi , Just an update. I have made the changes for this issue as suggested. But i have identified the output is not proper like before. seems like generate has some issue. its adding initial prompt with all the chunks. Will check and update on this. Also let me know if any existing issue going on this to your knowledge.

cc @kamilakesbi as @sanchit-gandhi is off 

are there any updates on this? or other ways you know of for pushing the model to more easily detect certain words using this pipeline?

c @ylacombe 

Hey @basicblueberrry136, thanks for your comment!
@sanchit-gandhi's review still has to be addressed before the next steps. Once it's done, I'll make another review! Hopefully it'll move fast!

I believe this is very helpful when used with the serverless inference API.

It seems that the serverless inference API uses the Transformers library to run models, and we cannot pass any parameter that has a type of tensor, as shown below:

```ts
const data = fs.readFileSync(filename);
const b64 = data.toString('base64');

const body = JSON.stringify({
    inputs: b64,
    parameters: {
        return_timestamps: true,
        generate_kwargs: {
            num_beams: 1,
            prompt_ids: [50362, 27338, 3763, 48022, 2257, 48022, 6784, 118, 25157, 1546, 15789, 23987, 5975, 17174, 28472, 25750, 6062, 1543],
        }
    }
});
```

It results in the following error:

```json
{
  "error": "unknown error",
  "warnings": [
    "There was an inference error: unknown error: list indices must be integers or slices, not NoneType"
  ]
}
```

If `initial_prompt` is added, we can pass the prompt as a string to the serverless inference API.

Hi, thanks for your work! Are there any updates on this?