Large audio chunking for the existing ASR pipeline
Author: anton-l
Pull request: yes

# What does this PR do?

This adds audio chunking with fixed-sized chunks as a first step to enabling audio streaming in ASR pipelines (ref: https://github.com/huggingface/transformers/pull/14250) 

In this iteration there's no ffmpeg streaming or VAD, just simple slicing of inputs with padding, so that we can review the general pipeline **from the modeling side**.
Here's an illustration of the sliding window approach used for iterating over chunks:
![chunking](https://user-images.githubusercontent.com/26864830/147242605-f32faf7b-f062-4cdc-9028-a356bf9df78e.png)

To see how this will roughly look for real-time inference (when we implement it on top), check out this (admittedly old) demo: https://huggingface.co/spaces/anton-l/youtube-subs-wav2vec/

Comments:
@Narsil sorry, I only now remembered about the work in [#14225](https://github.com/huggingface/transformers/pull/14225)!
Indeed, the `ChunkPipeline` API is much cleaner and I can adapt this PR to work with it.

Re: 2-4) totally agree, these points should be refactored as you suggest!
Re: 5) If I understand correctly, you suggest using `ChunkDataset/VADChunkDataset` only for offline datasets? Then we would still need to take parts of their chunking logic outside, to reuse them for streaming inputs


> @Narsil sorry, I only now remembered about the work in [#14225](https://github.com/huggingface/transformers/pull/14225)! Indeed, the `ChunkPipeline` API is much cleaner and I can adapt this PR to work with it.

It got autoclosed so even I struggled to find it again yesterday :)
> 
> Re: 2-4) totally agree, these points should be refactored as you suggest! 


> Re: 5) If I understand correctly, you suggest using `ChunkDataset/VADChunkDataset` only for offline datasets? Then we would still need to take parts of their chunking logic outside, to reuse them for streaming inputs

Actually, we might be able to make them interoperable too. 

Well I am mentionning `Dataset` but actually the pipeline works with any `generator` so we can use that for streaming too ! This is actually what `ffmpeg_microphone` does.

I like mentioning `Dataset` since a dataset has a fixed number of elements, meaning `tqdm` and the like can infer a nice progress bar and time estimates. But everything works quite the same with a `generator` except:
- `num_workers` cannot be used with values >1 (fetching from a generator from multiple threads is asking for trouble since you need to iterate on ALL objects on EVERY thread, even if you skip some on some threads, most likely the generator will already consume resources and time)
- No nice progress bar and time estimate with  `tqdm`.


So we could imagine something like:

```python
dataset = datasets.load_dataset(...)
vad_dataset = vad_cut(dataset,  threshold=5, ..)
chunk_dataset = chunk_audio(dataset, chunk_len_ms=200)
```

or 

```python
microphone_generator = ffmpeg_microphone(...)
for chunk in pipe(chunk_audio(microphone, chunk_len_ms=200)):
    print(chunk)
```

I am unsure it makes total sense and that we should make ALL of them interoperable and such, but there definitely could be a nice way to make those chunking iterator composable (just like `torchvision.transforms` can be for instance).
It would be a very nice thing indeed.

Thanks for the feedback @Narsil! I don't fully agree here  - happy to discuss this a bit more (also in a call). Maybe I'm also not seeing something here.

> I think it's a good intention PR, but IMHO we should probably refactor to pull everything out of the pipeline code for several reasons. I actually started work on #14250 in a similar fashion then scrapped everything. The reasons are
> 
> 1. it doesn't play nice with the auto batching /  `DataLoader` framework explaining some failing tests most likely:
> 
> ```python
> pipe = pipeline(..)
> for out in pipe(dataset, batch_size=32):
>     # do something with out
> ```
> 
> Using this allows users to adjust the batch_size relative to the hardware they have to maximize performance.
> 
> This unfortunately means:
> 
> * No loop in `_forward`.
> * No loop/batch in `preprocess`.
> 
> This PR could enable it back again: #14225 (it was the main reason I started this PR in the first place).

Here, I don't really know whether it plays nicely with auto batching or not. I agree that it is very important to make it work nicely with auto batching since I can see companies being interesting in transcribing tons of audio files offline and it would be nice to have that working fast.

However, for me it's at least equally important to make sure it's **very** simple for the user to transcribe a single audio file. The main applications for this feature in my opinion are:
- demo widget. Orgs on the hub will want to demo their models to clients, internally, etc... we have already seen demand for this
- should be easy to build a space with this feature to transcribe long audio files of e.g. Videos (YouTube, TED) on the fly

IMO those features are **more** important than auto batching / `Data Loader`. A necessary requirement to make chunking easy to use is that one doesn't have to wrap her/his audio file in some kind of data loader, wrapper, datasets, etc. I feel pretty strongly about making it possible to just do:

```python
transcription = asr("<path/to/audio/file>", chunking_length_in_s=2)
```

> 2. The arguments are not in `_sanitize_parameters` which means they will be only used in the initalization in the pipeline and not at call time. Since historically they were a lot of issues there, now all arguments are enabled in both which saves user sanity without figuring out where the arguments should be defined (look at other pipelines to see how they are implemented, its not a big change code wise, and the current doc is correct).

Agree. We can however easily change that.

> 3. `chunking`, `chunking_len`, `chuking_start_padding_len` and `chunk_end_padding_len` are not making sense independently.

>    For arguments, it is always easier for the user if there's no interaction between arguments and you can independently modify any of them. Here we could imagine having a single `chunking` that be either `None` or a triplet `(len, start_padding, stop_padding)` for instance. That would reduce a lot of confusion IMHO.

Disagree here. I really don't like tuple input arguments as one never knows which index of the tuple stands for what and this has to be looked up again in the docs every time someone uses the pipelines. We don't have many tuple args in `transformers` in general, but rather prefer "simple" args (in the config, function, etc...). 

I also think they are quite independent from each other - changing one arg "`chunking_len`" doesn't mean that `"chunk_end_padding_len"` has to be changed either. 

My 5 cents here are:
- 1.) Remove the `"chunking"` input arg. I don't like boolean flags either and I think whether the input shoudl be chunked or not should be controlled by `"chunking_len"`. If `"chunking_len"` is > 0, then no chunking else chunking.
- 2.) (nit) I would write out `len` to `length` and IMO `chunking_padding_left` is easier to understand then "...start..."

> 4. `chunking_start_padding_len` is expressed in number of samples which again depend on `sampling_rate` which might be different for different models, meaning quality might change if you use a different model. I would much prefer have `start_padding_len_ms` for instance since as a user it makes more sense to adjust in that space than in raw examples length space.

Agree. Nice observation!

> 5. If we want to add `webrtc` (which exists in #14250 ) then we have to add a whole bunch of new arguments, which would conflict with these added ones.

Not sure I fully agree here. Why would those new arguments conflict with `chunking_length`? IMO, `chunking_length` should default to either 0 or `None`, *i.e.* be disabled. In the future we could image a `vad="webrtc"` argument or `vad=WebRTCVAD("<all_necessary_args_here>")` argument and I don't really see why conflicts with `"chunking_length"` or chunking_length_left/right . `"chunking_length"` can still be used (and refer to the maximum allowed chunk length) and the other two arguments could simply be set to 0.

>    IMO it would make a whole more sense to instead choose the following form for users:
> 
> ```python
> pipe = pipeline(...)
> dataset = load_dataset(...)
> chunk_dataset = ChunkDataset(dataset, length_ms, start_pad_ms, end_pad_ms)
> 
> for chunk in pipe(dataset):
>     print(chunk)
> ```
> 
> In this way we could add `VADChunkDataset(dataset, threshold=2, frame_size_ms=20)` for instance quite orthogonally without cluttering the pipeline with a ton of arguments.
> 

This looks clean, but it's not easy to understand for the user. What if I just have a single long audio file that I want to transcribe? Do I first need to put it in some kind of `dataset` format? Users won't make that effort, they'll simple stop at this point. This relates to 1.) and here it's much more important to make this feature easy to try out instead of having something super extendable/general . For me pipelines always had the spirit of "2 lines of code is enough" and this starts to look much more complex for the user. 

Also given that already have a bunch of input arguments for pipelines ("generate()") takes 50+ input arguments, I don't really see a problem with adding many new arguments + we can also mark something as experimental and change later.

If you are very strongly against just adding input arguments @Narsil, maybe we can find a compromise where we over some kind of object `ChunkWithPadding` that is required to have a `chuck(...)` method and can wrap all kinds of inputs, *e.g.*:

```python
from transformers import pipeline, ChunkWithPadding

asr = pipeline("automatic_speech_recognition")

chunked_audio = ChunkWithPadding("path/to/audio/file", "<args>")

asr(chunked_audio)   # here inside we call `.chunk()` at some point
```

But I don't think that's very clean and I'd much rather prefer to do:

```python
from transformers import pipeline, ChunkWithPadding

asr = pipeline("automatic_speech_recognition", chunking_length_in_s=10)

asr("/path/to/audio")
```

> It also enables more complex stuff like `ffmpeg_microphone` where the audio samples actually overlap and some results are "temporary" (replaced with other chunks later).
> 
> It also enables fast feedback because you get the results as soon as they come in (without waiting when it's an hour long audio sample for instance).
> 
> The main drawback from this approach is that you don't have any information within `chunk` to know from which file it comes from or which chunk it is. It might be important to recreate the end result as a user. But IMHO, it seems easier to start passing those information through the pipeline so that they are available within `chunk` to allow the user to assemble chunks as they see fit.
> 
> I am happy to discuss anything that I might have overlooked in this analysis and why this PR might still be the right solution. Again this was also my first idea.

=> So to summarize, my by far biggest concern here is that the feature is to difficult to use for the user. I think we should think more about the user experience and not so much about making it super general, 100% clean from the inside. Pipelines are IMO the part of `transformers` we can and should absorb complexity so that the user has a **very** good user experience at the cost of maybe some ugly code inside `pipelines`. 
It would be nice to focus here on a first solution that works well and has a nice user experience before thinking about how this feature could conflict with a feature that will potentially added in the future. If I understand correctly the "padding-chunk" approach works well in all kinds of settings and is also very light weight. There are no necessary imports of other libraries etc...Webrtc seems to work only equally well, but is more data dependent (does it work well with noise, different language?!), it has an important dependency and an additional model that needs to be loaded. So for me (if the above is correct), it's pretty clear that we should focus on the padding chunking approach in a first step. We don't even know if the users will use this a lot or not. It's very good to also think about how this would pan out with a future WebRTC integration, but I also don't really see a problem here with the argument `"chunking_length_in_s"`.
Regarding the technical details, I'm not really sure how to solve this, but I also much rather prefer to add a small hack or a new design, etc... instead of forcing the user to load a single audio sample in some kind of dataset or generator.

Happy to jump on a call about this!


Pretty big but important clarification I didn't get when I read the PR:

The purple thing on the diagram is coming from the real audio, and the pipeline can cut on the green boundaries during decodings on `ids` tensor (before CTC). meaning we should pretty much exactly the exact decoding than when running the full audio (as long as the full chunk green + purple covers the whole word).

The name `padding` is slightly misleading to me, since it's usually called `stride` (at least in `question-answering` and vision). I imagined the purple part was supposed to be `zeros`, meaning it would help the edge of the green data but not nearly as well as with real data (and could be done outside of the pipeline, which would be much harder to do with real data, since once you do CTC you loose all those Green/purple information.

- `padding` : Fill in with zeros
- `stride`: Make overlapping data within samples

This approach could definitely work and be integrated in the pipeline (since CTC would make information be lost otherwise).

- It needs to be CTC-only: it's unlikely to be sound for generative audio models, since there's no 1-1 mapping of audio samples to IDs.
- There's actually probably sane defaults for the stride length since a word should rarely exceed 30s. And those samples do fit most user-grade GPU. We can also have something like `qa` where the `stride` is defined by default as `1/4 of max_length` if my memory serves correctly.

All-in-all. Current PR approach should work and be done like it is right now (and having the utils externally for VAD and the like in the other PR). We just need to enabling `ChunkPipeline` here to make it work properly with `batch_size` (merged since)

> Pretty big but important clarification I didn't get when I read the PR:
> 
> The purple thing on the diagram is coming from the real audio, and the pipeline can cut on the green boundaries during decodings on `ids` tensor (before CTC). meaning we should pretty much exactly the exact decoding than when running the full audio (as long as the full chunk green + purple covers the whole word).
> 
> The name `padding` is slightly misleading to me, since it's usually called `stride` (at least in `question-answering` and vision). I imagined the purple part was supposed to be `zeros`, meaning it would help the edge of the green data but not nearly as well as with real data (and could be done outside of the pipeline, which would be much harder to do with real data, since once you do CTC you loose all those Green/purple information.
> 
> * `padding` : Fill in with zeros
> * `stride`: Make overlapping data within samples
> 
> This approach could definitely work and be integrated in the pipeline (since CTC would make information be lost otherwise).
> 
> * It needs to be CTC-only: it's unlikely to be sound for generative audio models, since there's no 1-1 mapping of audio samples to IDs.
> * There's actually probably sane defaults for the stride length since a word should rarely exceed 30s. And those samples do fit most user-grade GPU. We can also have something like `qa` where the `stride` is defined by default as `1/4 of max_length` if my memory serves correctly.
> 
> All-in-all. Current PR approach should work and be done like it is right now (and having the utils externally for VAD and the like in the other PR). We just need to enabling `ChunkPipeline` here to make it work properly with `batch_size` (merged since)

Thanks a lot for summarizing everything here! After the call I very much agree that `stride_length_in_sec` is a better name indeed

@patrickvonplaten Could you review https://github.com/huggingface/transformers/pull/14250 too ? (it needs a rebase but I think it would be nice if both PRs are available roughly at the same time.

Let's do that.

Would be nice if we could make a short blog post showcasing how this feature works! Think we'll just need an hour long audio clip or so (maybe a clean speech from someone, *e.g.* US president) and then a couple of lines of code :-) 

Probably don't even need the blog post. Think an entry here: https://discuss.huggingface.co/ would be enough