Evaluation with HuggingFace Whisper
Author: qnxgy921
Pull request: no

### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.35.0
- Platform: Linux-5.15.0-88-generic-x86_64-with-glibc2.31
- Python version: 3.10.12
- Huggingface_hub version: 0.16.4
- Safetensors version: 0.3.1
- Accelerate version: 0.25.0.dev0
- Accelerate config:    not found
- PyTorch version (GPU?): 2.0.1+cu118 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): 0.7.0 (gpu)
- Jax version: 0.4.13
- JaxLib version: 0.4.13
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help?

@sanchit-gandhi
@gante


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

I was using the example evaluation code mentioned in [to](https://huggingface.co/openai/whisper-large) evaluating my own model.
I also used the datasets to load my own test dataset. 
The source code looks like below:
```
dss = load_dataset("audiofolder", data_dir="./uatdata", split="train", cache_dir="my/datasets")
for ds in dss:
    prompt_ids = prossor.get_prompt_ids(prompts, return_tensors="pt")
    input_features = processor(audio["array"], sampling_rate=audio["sampling_rate"], return_tensors="pt").input_features
    # generate token ids
    with torch.no_grad():
        predicted_ids = model.generate(input_features.to("cuda"), prompt_ids=prompt_ids)
    # decode token ids to text
    transcription = processor.decode(predicted_ids[0], skip_special_tokens=False)
    print(transcription)
```


### Expected behavior

1. The torch size of prompt_ids is around 200. I printed the prediction result. I found that the prompt_id would repeated with the next audio until until they occupy the results of all generations like:
```
TKOL2910_segment0.wav

 submitted cystectomy cyst urethral tumourappearance corpusprecedure colon straddle skin subcutis adjacent bilateral consistency macroscopically rectal medial uterus marginsresection inferior peritoneal bladder dye perforation intestine urinary multifocal synchronous lymph distal vulval biopsy myometrium extensive ovarian nipple invasion unifocal vaginal focality lateral radical tumour rectum fallopian procedure posterior tumournumber ovary identify length tme mesenteric THBSO tumoursite ureteral serosal resection node perivesical parametrial apr circumference margin completeness proximal organ cervical grossly longitudinal mesorectum dentate gross radial ureter anterior breast involved superior uterine axillary transverse CA ostial endometrial bisected anastomosis intestinal mesoappendiceal mucosa mesoappendix isthmic bicornually reflection ulcerated vascular pedicle donut

22AB3872 K040 skin left thigh

TKOL2910_segment1.wav

 submitted cystectomy cyst urethral tumourappearance corpusprecedure colon straddle skin subcutis adjacent bilateral consistency macroscopically rectal medial uterus marginsresection inferior peritoneal bladder dye perforation intestine urinary multifocal synchronous lymph distal vulval biopsy myometrium extensive ovarian nipple invasion unifocal vaginal focality lateral radical tumour rectum fallopian procedure posterior tumournumber ovary identify length tme mesenteric THBSO tumoursite ureteral serosal resection node perivesical parametrial apr circumference margin completeness proximal organ cervical grossly longitudinal mesorectum dentate gross radial ureter anterior breast involved superior uterine axillary transverse CA ostial endometrial bisected anastomosis intestinal mesoappendiceal mucosa mesoappendix isthmic bicornually reflection ulcerated vascular pedicle donut
 submitted cystectomy cyst urethral tumourappearance corpusprecedure colon straddle skin subcutis adjacent bilateral consistency macroscopically rectal medial uterus marginsresection inferior peritoneal bladder dye perforation intestine urinary multifocal synchronous lymph distal vulval biopsy myometrium extensive ovarian nipple invasion unifocal vaginal focality lateral radical tumour rectum fallopian procedure posterior tumournumber ovary identify length tme mesenteric THBSO tumoursite ureteral serosal resection node perivesical parametrial apr circumference margin completeness proximal organ cervical grossly longitudinal mesorectum dentate gross radial ureter anterior breast involved superior uterine axillary transverse CA ostial endometrial bisected anastomosis intestinal mesoappendiceal mucosa mesoappendix isthmic bicornually reflection ulcerated vascular pedicle donut

 vesular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nodular nod

```

   Is there anything wrong with my code?

2. Besides that, I also found that when I set `skip_special_tokens=True` when decode, the prompt_ids would not be masked.
```
transcription = processor.decode(predicted_ids[0], skip_special_tokens=True)
print(transcription)

TKOL2910_segment0.wav
submitted cystectomy cyst urethral tumourappearance corpusprecedure colon straddle skin subcutis adjacent bilateral consistency macroscopically rectal medial uterus marginsresection inferior peritoneal bladder dye perforation intestine urinary multifocal synchronous lymph distal vulval biopsy myometrium extensive ovarian nipple invasion unifocal vaginal focality lateral radical tumour rectum fallopian procedure posterior tumournumber ovary identify length tme mesenteric THBSO tumoursite ureteral serosal resection node perivesical parametrial apr circumference margin completeness proximal organ cervical grossly longitudinal mesorectum dentate gross radial ureter anterior breast involved superior uterine axillary transverse CA ostial endometrial bisected anastomosis intestinal mesoappendiceal mucosa mesoappendix isthmic bicornually reflection ulcerated vascular pedicle donut22AB3872 K040 skin left thigh
```

When I used transformers 4.30.2 , the result should look like:
```
TKOL2910_segment0.wav
22AB3872 K040 skin left thigh
```

Comments:
Hey @qnxgy921, could you possibly share the audio sample you're using so we can run the example end-to-end? Currently, it uses a local audio file, so we cannot reproduce the repeating prompt phenomenon. Happy to advise on how to fix the parameters here!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.