prompt_ids are included in distil-whisper response when using ASR pipeline()
Author: benniekiss
Pull request: no
Labels: name: Core: Pipeline

name: Good Second Issue

name: Audio

### System Info

- `transformers` version: 4.36.0.dev0
- Platform: macOS-14.1.1-arm64-arm-64bit
- Python version: 3.11.6
- Huggingface_hub version: 0.19.4
- Safetensors version: 0.4.0
- Accelerate version: 0.24.1
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.0 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@Narsil @sanchit-gandhi

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```
import torch
from transformers import AutoModelForSpeechSeq2Seq, AutoProcessor, pipeline
from datasets import load_dataset


device = "cuda:0" if torch.cuda.is_available() else "cpu"
torch_dtype = torch.float16 if torch.cuda.is_available() else torch.float32

model_id = "distil-whisper/distil-large-v2"

model = AutoModelForSpeechSeq2Seq.from_pretrained(
    model_id, torch_dtype=torch_dtype, low_cpu_mem_usage=True, use_safetensors=True
)
model.to(device)

processor = AutoProcessor.from_pretrained(model_id)

pipe = pipeline(
    "automatic-speech-recognition",
    model=model,
    tokenizer=processor.tokenizer,
    feature_extractor=processor.feature_extractor,
    max_new_tokens=128,
    chunk_length_s=15,
    batch_size=16,
    torch_dtype=torch_dtype,
    device=device,
## MODIFIED:
    decoder_kwargs={
        "skip_special_tokens": True
    }
)

## MODIFIED:
whisper_prompt = "This is a sample prompt before a transcription."
prompt_ids = pipe.tokenizer.get_prompt_ids(whisper_prompt, return_tensors='pt')

dataset = load_dataset("distil-whisper/librispeech_long", "clean", split="validation")
sample = dataset[0]["audio"]

result = pipe(
    sample,
## MODIFIED:
    generate_kwargs={
       "prompt_ids": prompt_ids
    }
)
print(result["text"])
```

### Expected behavior

I would expect that the prompt tokens would be stripped from the response given by `pipeline()`. 
It is possible to strip the tokens by using `skip_special_tokens=True` with `decode()`, however, it is not possible to pass this variable via `decoder_kwargs` to `pipeline()`, unless I am mistaken.

Comments:
I believe this part of the code, [tokenization_whisper.py#L853](https://github.com/huggingface/transformers/blob/dc68a39c8111217683bf49a4912d0c9018bab33d/src/transformers/models/whisper/tokenization_whisper.py#L853), would have to be changed to include an argument containing the prompt and an argument to remove the prompt, as well as an `if...else` block to strip the prompt from the response.

Correcting the ping to cc @sanchit-gandhi. 
The addition of the `tokenizer_kwargs` to the call is planned but no-one is working on it at the moment. You cannot do this yet indeed. 

I think the `prompt_ids` need to be stripped from the `token_ids` before we compute the longest common sub-sequence in the chunking algorithm. Otherwise, we try to match the generated token ids on the right with the prompt input ids on the left (inevitably going to mis-match!). Opened a PR for this here: #27836

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

should I close this issue since there is an open PR, or should I keep the issue open until the PR is merged?

Let's keep it open until the PR is merged!

Just a quick update -- I haven't tested this PR with an updated main yet because currently, prompt_ids are no longer returned with long-form transcription; however, they are still returned with short-form transcription.
transformers v 4.38.0dev0

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

closing as the PR was merged

Was this closed by a different PR? #27836 is still open.