ASR pipeline does not work with openai/whisper on current master
Author: niedakh
Pull request: no

### System Info

transformers @ git+https://github.com/huggingface/transformers.git@b651efe59ea506d38173e3a60a4228e7e74719f9
python 3.6
Standard AWS Ubuntu Deep Learning AMI (Ubuntu 18.04) Version 30.0

### Who can help?

@Narsil @anton-l @sanchit-gandhi @patrickvonplaten 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

To reproduce run the following code, from asr pipeline example and whisper:

```python
from datasets import load_dataset
from transformers import pipeline

pipe = pipeline(model="openai/whisper-large")
ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
output = pipe(ds[0]['file'], chunk_length_s=30, stride_length_s=(4, 2))
```

yields:

```
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-13-efceed64cd5c> in <module>
----> 1 output = pipe(ds[0]['file'], chunk_length_s=30, stride_length_s=(4, 2))

~/venv38/lib/python3.8/site-packages/transformers/pipelines/automatic_speech_recognition.py in __call__(self, inputs, **kwargs)
    181                         `"".join(chunk["text"] for chunk in output["chunks"])`.
    182         """
--> 183         return super().__call__(inputs, **kwargs)
    184 
    185     def _sanitize_parameters(self, **kwargs):

~/venv38/lib/python3.8/site-packages/transformers/pipelines/base.py in __call__(self, inputs, num_workers, batch_size, *args, **kwargs)
   1072             return self.iterate(inputs, preprocess_params, forward_params, postprocess_params)
   1073         else:
-> 1074             return self.run_single(inputs, preprocess_params, forward_params, postprocess_params)
   1075 
   1076     def run_multi(self, inputs, preprocess_params, forward_params, postprocess_params):

~/venv38/lib/python3.8/site-packages/transformers/pipelines/base.py in run_single(self, inputs, preprocess_params, forward_params, postprocess_params)
   1093     def run_single(self, inputs, preprocess_params, forward_params, postprocess_params):
   1094         all_outputs = []
-> 1095         for model_inputs in self.preprocess(inputs, **preprocess_params):
   1096             model_outputs = self.forward(model_inputs, **forward_params)
   1097             all_outputs.append(model_outputs)

~/venv38/lib/python3.8/site-packages/transformers/pipelines/automatic_speech_recognition.py in preprocess(self, inputs, chunk_length_s, stride_length_s)
    260             # Currently chunking is not possible at this level for `seq2seq` so
    261             # it's ok.
--> 262             align_to = self.model.config.inputs_to_logits_ratio
    263             chunk_len = int(round(chunk_length_s * self.feature_extractor.sampling_rate / align_to) * align_to)
    264             stride_left = int(round(stride_length_s[0] * self.feature_extractor.sampling_rate / align_to) * align_to)

~/venv38/lib/python3.8/site-packages/transformers/configuration_utils.py in __getattribute__(self, key)
    252         if key != "attribute_map" and key in super().__getattribute__("attribute_map"):
    253             key = super().__getattribute__("attribute_map")[key]
--> 254         return super().__getattribute__(key)
    255 
    256     def __init__(self, **kwargs):

AttributeError: 'WhisperConfig' object has no attribute 'inputs_to_logits_ratio'
```

### Expected behavior

I would've expected to obtain the transcript in `output`.

Comments:
cc @ArthurZucker 

I will have a look! But chunking is not supported yet with whisper. (Should take care of it next week) 
Normally a warning should pop instead of an error

@ArthurZucker are we sure `Whisper` can handle chunking ? 

> Whisper is not a CTC model meaning that chunking as shown in Nico's [blog](https://huggingface.co/blog/asr-chunking) does not work.

from internal conversation.

Happy to jump into a design call to discuss whether we can do it or not. 

Not being CTC means it's harder to handle the boundaries. Boundaries at silence are sort of OK, but unfortunately can never really a *complete* solution (because you can never be sure you're going to get a silence, and you MUST be able to handle chunking regardless).
This might be deemed acceptable in whisper btw, but when we checked for regular models, the regular silence detection was not good enough to be ran automatically (meaning you have to tune settings always to get decent silence results with most silence detectors)

Really sorry about my miss-communication. The chunking that will be supported is different from CTC. Let's organize a call to speak in more details about that 😉 
The goal would be to be able to specify a chunk length and stride length (if people want to customize it) but default Whisper has its own parameters. Let's talk more about that when we call 🤗

Would also be interested to hear more about how chunking will be supported for the Whisper ASR pipeline! 😁

Related to this: is there a way to avoid the transcription being cut off too early with a HF ASR pipeline? It seems the ASR pipeline will only transcribe the 1st section if we have a longer audio file with silence in between. 

Hi @CarloLepelaars 

It's actually quite challenging to do chunking with whisper. The reason is that the suggested way by OpenAI needs to run the inference on the first 30s before being able to run inference on the next 30s starting at 30s - X. X depends on the output of the first run.

This violates an important property of a pipeline, which is that the generations shouldn't depend on each other (in order to enable batching).

That doesn't mean it's impossible, but the stiching back of actual predictions becomes hairy:
 - We have no control where the timestamps are created, and we can't force them to appear within the strides.
 - It also require an extremely custom `LogitsProcessor` to "force"  timestamp tokens to appear. 
 
 For the audio being cut off, would you rather have an error being thrown ? Maybe @ArthurZucker has better ideas what we should do when the audio is too long.

Thanks @Narsil ! 
For long audio, we can just enable the chunking without `timestamp` prediction. Though the results won't be extremely good, I remember attempting this (ultra naive way, with no `stride` )  and it gave pretty decent outputs : 

```python 
"""
Je mappelle Claude. Je decoupe plouf. Let's just try it again. Je mappelle Claude. Je te plie mlu. Huh. It's not quite what I'm saying. Really? Sounds exactly the same to me. It does? Really? Yeah. All right, let's just try it again. Really listen.
 Okay. Je mappelle Claude. Je te flou... flie. Oh, mon Dieu. Oh, de fouf. Je mappelle Claude. Je te call blue. No! Okay, maybe if we just break it down. Okay, let's just... let's try it one syllable at a time. Okay, so repeat after me. Je... Je... Ma... Ma... Pelle.
 Great! Okay, faster. Je m'mappelle. Je m'mappelle. Me poo poo! It's too hard. I can't teach you. What are you doing? I have to go before I put your head through a wall. Don't go! Don't go! I need you! My audition is tomorrow! Jableau Blanc! Mille lapis! Au Blanc! Pou!
"""
```
For this clip : https://www.youtube.com/watch?v=H3dToD7_ATU , which seem like apart from maj, is extremely good! 


@ArthurZucker What about erroring out when the input audio is too long ?

I'd rather add a warning saying that the audio will be automatically cropped! WDYT? 

IMO error is better here.

@Narsil if there is a chance to run Whisper's silence detection + chunking mechanism in a pipeline I think this would be very useful/impactful 

> I will have a look! But chunking is not supported yet with whisper. (Should take care of it next week)
> Normally a warning should pop instead of an error

Has this been implemented? Where can I check the upgrades for when it is functional? 

(I understand it is not an easy task, just wanted to make sure that I have the tools to find out about the implementation when it is available).

Hey, here is one of the PR: #20104

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Resolved in https://github.com/huggingface/transformers/pull/20104