whisper prompt_ids gets added to decoded predictions
Author: M-Ali-ML
Pull request: no
Labels: name: Good Second Issue

### System Info

duplicated the issue in google collab:

- `transformers` version: 4.35.2
- Platform: Linux-5.15.120+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.1
- Accelerate version: 0.25.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0+cu118 (True)
- Tensorflow version (GPU?): 2.14.0 (True)
- Flax version (CPU?/GPU?/TPU?): 0.7.5 (gpu)
- Jax version: 0.4.20
- JaxLib version: 0.4.20
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: no



### Who can help?

@sanchit-gandhi

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

created a collab reproduction of the issue [here](https://colab.research.google.com/drive/1v0NaGrw1NVbrj14Y8LTmJKMy8zVDJf1u?usp=sharing)

here is the main code snippet:


```python
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "openai/whisper-large-v3"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)
prompt_text = "Mr. kuilter"
prompt_ids = processor.get_prompt_ids(prompt_text, return_tensors="pt")

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=30,
    batch_size=16,
    return_timestamps=True,
    torch_dtype=torch_dtype,
    device=device,
    generate_kwargs={
       "prompt_ids": prompt_ids
       }
)

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]
```



### Expected behavior

without prompting the output text is:

>  Mr. Quilter is the apostle of the middle classes, and we are glad to welcome his gospel. Nor is Mr. Quilter's manner less interesting than his matter. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. Linnell's pictures are a sort of Upguards and Adam paintings, and Mason's exquisite idylls are as national as a jingo poem. Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, Next man!

with prompting the output text is:

> Mr. kuilter Mr. Kuilter is the apostle of the middle classes, and we are glad to welcome his gospel. Nor is Mr. Kuilter's manner less interesting than his matter. He tells us that at this festive season of the year, with Christmas and roast beef looming before us, similes drawn from eating and its results occur most readily to the mind. He has grave doubts whether Sir Frederick Leighton's work is really Greek after all, and can discover in it but little of rocky Ithaca. Lynell's pictures are a sort of Upgards and Adam paintings, and Mason's exquisite idylls are as national as a django poem. Mr. Burkett Foster's landscapes smile at one much in the same way that Mr. Carker used to flash his teeth. And Mr. John Collier gives his sitter a cheerful slap on the back before he says, like a shampooer in a Turkish bath, next man.



The prompt did effect the generated decoded predictions changing all instance of "Quilter" to "kuilter" as intended, however, the output result contain the prompt "Mr Kuilter" in the beginning of the decoded predictions (an extra one coming from the prompt).
I'm not sure why, but prompt is getting added to the prediction. That shouldn't be the case as shown in [openAI cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_prompting_guide.ipynb) .

Comments:
Duplicate of #27594, will be closed by https://github.com/huggingface/transformers/pull/27836.

Closed by https://github.com/huggingface/transformers/pull/27836!