Correct way for Wav2vec2 feature extraction from huggingface like Fairseq
Author: hungdinhxuan
Pull request: no
Labels: name: Audio

### System Info

- `transformers` version: 4.36.2
- Platform: Linux-6.2.0-37-generic-x86_64-with-glibc2.35
- Python version: 3.10.13
- Huggingface_hub version: 0.20.2
- Safetensors version: 0.4.1
- Accelerate version: not installed
- Accelerate config: not found
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: NVIDIA RTX 4090
- Using distributed or parallel set-up in script?: No

### Who can help?

_No response_

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

```
import fairseq
from transformers import Wav2Vec2ForPreTraining, Wav2Vec2Config
import torchaudio

model_file = 'wav2vec_small.pt'
model, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([model_file])
original = model[0]
waveform, _ = torchaudio.load('MMSTTS_ara_000008.wav')
reference = original(waveform, mask=False, features_only=True)['x']
model_name = 'facebook/wav2vec2-base'
config = Wav2Vec2Config.from_pretrained(model_name)
w2vhf = Wav2Vec2ForPreTraining.from_pretrained(model_name, config=config)
res = w2vhf(waveform, attention_mask=None, output_hidden_states = True).hidden_states[-1]

torch.testing.assert_close(res, reference)
```

Error:
```
AssertionError: Tensor-likes are not close!

Mismatched elements: 155126 / 155136 (100.0%)
Greatest absolute difference: 5.448995590209961 at index (0, 140, 96) (up to 1e-05 allowed)
Greatest relative difference: 89616.765625 at index (0, 6, 144) (up to 1.3e-06 allowed)
```

### Expected behavior

I am using pre-trained wav2vec-base model download from fairseq GitHub. I expected that the pre-trained model provided by huggingface is also the same as fairseq. It means feature extractions from fairseq and hugging face should be as close as possible. What am I doing wrong, or is my feature extraction using Wav2Vec2ForPreTraining not correct? Pre-trained model provides by huggingface with the Pre-trained model from Fairseq are the same? Are there any differences between their model architectures?

Comments:
cc @sanchit-gandhi @ylacombe 

Gentle ping @sanchit-gandhi @ylacombe 

Another ping @sanchit-gandhi @ylacombe

Hey @hungdinhxuan, note that you need to use the wav2vec2 feature extractor before passing your input to the model. Find an example on the docs [here](https://huggingface.co/docs/transformers/model_doc/wav2vec2#transformers.Wav2Vec2ForPreTraining.forward.example)!

You can notice [here](https://github.com/huggingface/transformers/blob/539ed75d501f9af87b28dc0325b8d2f62b791cd9/src/transformers/models/wav2vec2/feature_extraction_wav2vec2.py#L227-L235) that the feature extractor can normalize the input audio waveform and that according to the [config of the wav2vec2 checkpoint](https://huggingface.co/facebook/wav2vec2-base/blob/0b5b8e868dd84f03fd87d01f9c4ff0f080fecfe8/preprocessor_config.json#L2) you're using, normalizing is necessary!

I hope that it helps!

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.