Add numpy alternative to FE using torchaudio
Author: ylacombe
Pull request: yes

# What does this PR do?

Following on from #26182, which ported `torchaudio.compliance.kaldi.fbank` to `numpy` in `audio_utils`, this PR aims to enable the use of numpy porting in previous Feature Extractors (AST and SpeechToText) that used `torchaudio`.  It was discussed [here](https://github.com/huggingface/transformers/pull/26182#pullrequestreview-1631301297).

This serves two purposes:
1. to give some examples of how to use `audio_utils` instead of `torchaudio` for future Feature Extractors
2. the possibility of removing torchaudio altogether in the future.

A next step would be to port `audio_utils` to torch, which might be faster (cc @sanchit-gandhi), but this is still open to discussion. Is this really relevant? And will it be really faster? 

cc @ArthurZucker and @sanchit-gandhi 

Comments:
_The documentation is not available anymore as the PR was closed or merged._

Hey @ArthurZucker and @sanchit-gandhi, thanks for the review!

However, I'm not sure about what you meant [here](https://github.com/huggingface/transformers/pull/26339#pullrequestreview-1643752078):

> Looks good to me, aligned with @sanchit-gandhi on not adding the np support

And [here](https://github.com/huggingface/transformers/pull/26339#discussion_r1336885695):

> I'm fine with adding a comment somewhere or a section in the doc to not lose the info on how to use numpy to get the same results as torchaudio for futur references when we'll improve or numpy port!

@sanchit-gandhi seems to be in favor of removing torchaudio support to only focus on the numpy port here, whereas @ArthurZucker seems to be in favor on not adding the numpy support.

Maybe I misunderstood the comments here! Thanks for your help!


Sorry I was confused! I agree that we should remove the old code, but worried about the performance issue, since we had to re introduce torch STFT for Whisper for example.  (Performance wise backward compatible)

I've made a quick benchmark, on AST, with results here:
![image](https://github.com/huggingface/transformers/assets/52246514/473f69d9-8b6a-447f-a82d-4b9590668b06)

Basically, torchaudio is at least 19 faster than the numpy porting. If I haven't made any mistake in my benchmark, I'll be strongly in favor of keeping torchaudio compatibility.

WDYT @ArthurZucker and @sanchit-gandhi ? Can you also take a quick look at the benchmark code to make sure that my results are correct (or redirect me to an expert at HF haha) ?  

For reference, here is the benchmark code:
```python
from datasets import load_dataset
import pytest
from transformers import ASTFeatureExtractor

ds = load_dataset("hf-internal-testing/librispeech_asr_dummy", "clean", split="validation")
speech_samples = ds.sort("id").select(range(64))[:64]["audio"]
speech_samples = [x["array"] for x in speech_samples]


def torchaudio_unbatch():
    fe = ASTFeatureExtractor(use_torchaudio=True)
    
    for sample in speech_samples:
        input_features = fe(sample, padding=True, return_tensors="pt")

def np_unbatch():
    fe = ASTFeatureExtractor(use_torchaudio=False)
    
    for sample in speech_samples:
        input_features = fe(sample, padding=True, return_tensors="pt")

def torchaudio_batch_8():
    fe = ASTFeatureExtractor(use_torchaudio=True)
    
    for i in range(0,len(speech_samples),8):
        samples = speech_samples[i:i+8]
        input_features = fe(samples, padding=True, return_tensors="pt")

def np_batch_8():
    fe = ASTFeatureExtractor(use_torchaudio=False)
    
    for i in range(0,len(speech_samples),8):
        samples = speech_samples[i:i+8]
        input_features = fe(samples, padding=True, return_tensors="pt")

@pytest.mark.benchmark(
    min_rounds=5, disable_gc=True, warmup=False
)
def test_torchaudio_unbatch(benchmark):
    benchmark(torchaudio_unbatch)

@pytest.mark.benchmark(
    min_rounds=5, disable_gc=True, warmup=False
)
def test_torchaudio_batch_8(benchmark):
    benchmark(torchaudio_batch_8)


@pytest.mark.benchmark(
    min_rounds=5, disable_gc=True, warmup=False
)
def test_np_unbatch(benchmark):
    benchmark(np_unbatch)

@pytest.mark.benchmark(
    min_rounds=5, disable_gc=True, warmup=False
)
def test_np_batch_8(benchmark):
    benchmark(np_batch_8)

```


For future reference, here is the same benchmark with `Speech2TextFeatureExtractor`:
Previous conclusions still hold:
![image](https://github.com/huggingface/transformers/assets/52246514/95c50e94-6390-4ae4-8f83-0668c916a028)


It's also possible that we can optimize our `audio_utils.py`, WDYT?

Alright that's quite a significant difference - this probably requires overhauling the `audio_utils` file as you've suggested (use `torch`/`torchaudio` if available, **or** see where our numpy implementation is bottlenecked and try to improve it here).

Hey @sanchit-gandhi, thanks for the review here!

> My only thought is that maybe we should overhaul audio_utils.py with these changes, rather than do the if/else in the feature extraction code? 

We'd have to create a `fbank` method to `audio_utils` which would create `mel_filters` and `window` on-the-fly in that case right ?  (with hindsight, it doesn't matter much since creating `mel_filters` and `window isn't the bottleneck here)
 
In any case, I'd rather refactor that in another PR, which would maybe add the `torch` correspondence for every possible case in `audio_utils` 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.