SequenceFeatureExtractor does not support bfloat16
Author: anferico
Pull request: no
Labels: name: Audio

### System Info

- `transformers` version: 4.37.2
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.14
- Huggingface_hub version: 0.22.2
- Safetensors version: 0.4.2
- Accelerate version: 0.21.0
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.1.2+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: yes
- Using distributed or parallel set-up in script?: yes (Deepspeed)

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. Use the [SequenceFeatureExtractor.pad()](https://github.com/huggingface/transformers/blob/v4.37.2/src/transformers/feature_extraction_sequence_utils.py#L52) function to pad some PyTorch tensors with `dtype=torch.bfloat16`
2. Watch your code fail as `pad()` tries to convert your tensors to NumPy before padding them, which is not possible for bfloat16 tensors

### Expected behavior

Padding happens smoothly regardless of the tensors' data type, which can be achieved by not converting them to NumPy before padding. 

Comments:
Could you share a reproducer?"

@ArthurZucker sure, here it is:
```python
import torch
from transformers import AutoProcessor


fake_audio_sr = 16000
fake_audio_duration_in_seconds = 4.55
fake_audio_len = int(fake_audio_sr*fake_audio_duration_in_seconds)
fake_audio = torch.rand((fake_audio_len,), dtype=torch.bfloat16)

audio_processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
audio_processor(
    fake_audio,
    sampling_rate=fake_audio_sr,
    return_tensors="pt",
    padding=True,
)
```

which gives:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/models/wav2vec2/processing_wav2vec2.py", line 94, in __call__
    inputs = self.feature_extractor(audio, *args, sampling_rate=sampling_rate, **kwargs)
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/models/wav2vec2/feature_extraction_wav2vec2.py", line 199, in __call__
    padded_inputs = self.pad(
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/feature_extraction_sequence_utils.py", line 178, in pad
    processed_features[key] = [to_numpy(v) for v in value]
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/feature_extraction_sequence_utils.py", line 178, in <listcomp>
    processed_features[key] = [to_numpy(v) for v in value]
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/utils/generic.py", line 283, in to_numpy
    return framework_to_numpy[framework](obj)
  File "/path/to/my/conda_env/lib/python3.10/site-packages/transformers/utils/generic.py", line 268, in <lambda>
    "pt": lambda obj: obj.detach().cpu().numpy(),
TypeError: Got unsupported ScalarType BFloat16
```

@ArthurZucker any updates on this?

cc @kamilakesbi here I won't have the bandwidth to dig!

Gentle ping @kamilakesbi 

Hi @anferico, 

Thank you for submitting this issue! 

The `FeatureExtractor` is designed to take numpy arrrays as inputs (here `fake_audio ` will be indeed converted to a numpy array) and output tensors, as indicated in the [doc](https://huggingface.co/docs/transformers/en/model_doc/wav2vec2#transformers.Wav2Vec2FeatureExtractor.__call__.raw_speech).  

--> This design choice significantly simplifies the feature extraction code, as it allows us to work primarily with NumPy arrays, eliminating the need to check whether the input is in NumPy or Torch format.

Following this reasoning, it is not necessary to handle torch.bfloat16 inputs directly in the FeatureExtractor. Instead, you can downcast the output of the feature extractor to bfloat16 if required: 

```python
import torch
import numpy as np
from transformers import AutoProcessor

fake_audio_sr = 16000
fake_audio_duration_in_seconds = 4.55
fake_audio_len = int(fake_audio_sr*fake_audio_duration_in_seconds)

# numpy in
fake_audio = np.random.randn((fake_audio_len))

audio_processor = AutoProcessor.from_pretrained("facebook/hubert-large-ls960-ft")
input_values = audio_processor(
    fake_audio,
    sampling_rate=fake_audio_sr,
    return_tensors="pt",
    padding=True,
).input_values

# torch out
input_values = input_values.to(torch.bfloat16)
```

Hope it will help you! 

Makes sense, thanks a lot! Closing