AutomaticSpeechRecognitionPipeline throws dict key error even with the correct keys
Author: ghost
Pull request: no

ValueError: When passing a dictionary to AutomaticSpeechRecognitionPipeline, the dict needs to contain a "raw" key containing the numpy array representing the audio and a "sampling_rate" key, containing the sampling_rate associated with that array

reproduced by using the following code:

```
import torch
from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
  "automatic-speech-recognition",
  model="openai/whisper-small.en",
  chunk_length_s=30,
  device=device,
)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]

prediction = pipe(sample)["text"]

# we can also return timestamps for the predictions
prediction = pipe(sample, return_timestamps=True)["chunks"]

```
versions: 
torch==1.13.1
transformers==4.26.0
datasets==2.9.0



Comments:
cc @Narsil and @sanchit-gandhi 

This is just because the `sample` is consumed when passed to the pipeline:

```python
import torch
from transformers import pipeline
from datasets import load_dataset

device = "cuda:0" if torch.cuda.is_available() else "cpu"

pipe = pipeline(
  "automatic-speech-recognition",
  model="openai/whisper-small.en",
  chunk_length_s=30,
  device=device,
)

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
sample = ds[0]["audio"]

prediction = pipe(sample.copy())["text"] # <------------CHANGE HERE

# we can also return timestamps for the predictions
prediction = pipe(sample, return_timestamps=True)["chunks"]
```

This should work.

@sgugger we could remove that by cloning everything ourselves, but it forces a copy of the entire audio array when passed to the pipeline. We have even more subtle modifications where we don't copy, but we do need to pass *some* keys for live inference (used by the API at least) where we pass extra keys as-is to the caller so it can know how to handle the temporary results (since pipeline is stateless it's cumbersome to deal with those by other means).

Hey @dsingal0!

Looks like this code snippet was taken from the Whisper small.en README example which I added last week: https://huggingface.co/openai/whisper-small.en#long-form-transcription

I've updated the model README with @Narsil's fix: https://huggingface.co/openai/whisper-small.en/commit/d34e5b8002f2524cb84680607caa2f802de266cd (and all other Whisper model READMEs accordingly)

Feel free to open issues/PRs on the Hugging Face Hub if a code example doesn't look right!

With regards to `pipeline`, this behaviour is potentially a bit confusing coming from the `model`/`processor` approach, since this way does not consume the input dict, allowing the user to re-use inputs as they wish. This is ok with me provided it's suitably well explained in the docs!

I think it would be more elegant if we had a copy-free approach in the processing that did not consume the audio inputs as we currently do (if feasible):
https://github.com/huggingface/transformers/blob/3b9a1dc13209d0cab347bf2363d18963cc3f9194/src/transformers/pipelines/automatic_speech_recognition.py#L447

It's exactly as how I explained above we have 3 choices:


- `consume` (current behavior) this makes samples non reusable. IMO the best choice since reusing is only likely to be used while exploring.
- `copy`. This makes an extra copy of the audio. On small files it doesn't matter that much, but way too costly for hour long audio files.
- `not-passthrough`. Do not pass extra keys around (like `partial` during live microphone inference). This makes this particular use case quite hard to work with (because of the statelessness of pipeline and it would be a breaking change).

Sure, thanks for clarifying! Happy to stick with `consume` in this case!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.