`prompt_ids` does not seem to work with `repetition_penalty`
Author: kiyosumaeda
Pull request: no

### System Info

transformers== 4.30.0.dev0

### Who can help?

@connor-henderson @sanchit-gandhi 

Hi,
Thank you for supporting prompt for inferencing with whisper (https://github.com/huggingface/transformers/pull/22496)! I've used this feature for Japanese audio files to transcribe rare words (e.g., proper nouns and domain specific words), and found it does work, that is, prompt improves WER when we set token list of proper nouns as `prompt_ids`.
And I also succeeded in reducing repetition in whisper transcript with `repetition_penalty`. Whisper sometimes meaninglessly repeat words without `repetition_penalty`.
However, when I set both `prompt_ids` and `repetition_penalty`, prompt does not work well. I suspect `repetition_penalty` might suppress tokens specified as `prompt_ids`.

Could you please check the above behavior?

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
base_model_name_or_path = 'openai/whisper-large-v2'
model = WhisperForConditionalGeneration.from_pretrained(
    base_model_name_or_path,
    load_in_8bit=True,
    device_map='auto',
)

language = 'ja'
task = 'transcribe'
model = PeftModel.from_pretrained(model, str(model_dir)) # I currently use AdaLoRA-tuned model, but this is reproducible with pretrained large model.
tokenizer = WhisperTokenizer.from_pretrained(base_model_name_or_path, language=language, task=task)
processor = WhisperProcessor.from_pretrained(base_model_name_or_path, language=language, task=task)
feature_extractor = processor.feature_extractor
forced_decoder_ids = processor.get_decoder_prompt_ids(language=language, task=task)
pipe = AutomaticSpeechRecognitionPipeline(model=model, tokenizer=tokenizer, feature_extractor=feature_extractor)

prompt_text = 'prompt text'
prompt_ids = processor.get_prompt_ids(prompt_text)

with torch.cuda.amp.autocast():
    outputs = pipe(
        str(audio_file_path), # I use in-house audio files
        generate_kwargs={
            'forced_decoder_ids': forced_decoder_ids,
            'prompt_ids': prompt_ids,
            'num_beams': 4,
            'repetition_penalty': 2.0, # I change this value (1.0 ~ 2.0)
        },
        max_new_tokens=255,
    )
    print(outputs)
```

### Expected behavior

when `repetition_penalty=1.0`, prompt works.
As I gradually increase `repetition_penalty`, words I set as prompt will not appear in transcripts.

Comments:
Hey @kiyosumaeda! Cool to see that you're using PEFT to fine-tune Whisper and that prompting is working with the PEFT model - this is nice validation that the model doesn't exhibit catastrophic forgetting, since we don't fine-tune on the prompting task. Note that you can get quite considerable speed-ups to inference time by running the model in fp16 instead of int8 (see https://github.com/huggingface/peft/discussions/477#discussion-5213394) - this also seems to drastically reduce Whisper's propensity to hallucinate. Could you try this first as a step?

Hi @sanchit-gandhi! Thank you for the feedback and suggestion.
I will try fp16 for inference to see whether prompting still works and hallucination is reduced.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Did you have any luck here @kiyosumaeda?

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

I suggest that you can set 'encoder_repetition_penalty' = 'repetition_penalty', thus repetition_penalty wouldn't impact tokens in prompts.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.