Wav2Vec2
Author: patrickvonplaten
Pull request: yes
Labels: name: PR for Model Addition

# What does this PR do?

Adds Wav2Vec2 from https://github.com/pytorch/fairseq/blob/master/examples/wav2vec/README.md

This PR adds the wav2vec2 Acoustic model to Transformers. The model is different from "conventional" transformer
models since it classifies a raw waveform input (float array) into logits. Therefore the `Wav2Vec2Tokenizer` behaves quite differently from usual tokenizers in that it only pads an input instead of encoding it to token ids.

The fully functional model should be added in three steps:

1) Add the Acoustic model ready to be used for inference (This PR)
2) Add fine-tuning + pertaining functionality to the model (Next PR)
3) Add an example script showing how Wav2Vec2 can be used with a language model
4) Add an Encoder/Decoder version of Wav2Vec2.

# Usage

One can play around with a quick demo here: https://colab.research.google.com/drive/1xaVKQ739-ue0v8IuMZoMzOFc4-NlGQyd?usp=sharing and some usage examples as described on the model cards:
https://huggingface.co/models?filter=wav2vec2

# Review
In this PR, no training functionality is added to the model. Because this is quite complex for Wav2Vec2 this will be done in a follow up PR. 
It would be great, if we can however already merge this first part which allows to use Wav2Vec2 for inference.
The tokenizer is quite different, so it would be great if @thomwolf @n1t0 can also take a look here.
Since the model expects the raw waveform signal as an input, the name `input_ids` is changed to `input_values` standing for "a tensor of float values" - would be great if you can check this @LysandreJik @sgugger @thomwolf. 

# Done:

- [x] load pretrained weight into model
- [x] make sure forward pass yields equal outputs
- [x] successful transcription
- [x] add tokenizer
- [x] Think about how to add the two different architectures `Wav2Vec 2.0 Large (LV-60)`/`Wav2Vec 2.0 Large (LV-60) + Self Training` is different from `Wav2Vec 2.0 Large`/`Wav2Vec 2.0 Base` (layer_norm is switched and no group norm is used)
- [x] add model tests
- [x] add tokenizer tests
- [x] add docstring
- [x] clean config

# Future TODO:
- [ ] Add PreTraining & Fine-Tuning to model
- [ ] Add Encoder Decoder model / CTC decoding



Comments:
hi @patrickvonplaten thank you for creating this PR and converting some model from original to test it

i want to test by convert XLSR [model](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec ) using your `convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py`

by following command  `python convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py --checkpoint_path /content/xlsr_53_56k.pt --pytorch_dump_folder_path huggingface_model`

i got error 

```
File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 147, in convert_wav2vec2_checkpoint
    [checkpoint_path], arg_overrides={"data": dict_path}
  File "/usr/local/lib/python3.6/dist-packages/fairseq/checkpoint_utils.py", line 279, in load_model_ensemble_and_task
    state = load_checkpoint_to_cpu(filename, arg_overrides)
  File "/usr/local/lib/python3.6/dist-packages/fairseq/checkpoint_utils.py", line 231, in load_checkpoint_to_cpu
    setattr(args, arg_name, arg_val)
AttributeError: 'NoneType' object has no attribute 'data'
```

do i need to specify the --dict_path argument...if so ..where can i get them ? thanks

[this issue](https://github.com/pytorch/fairseq/issues/3050) seems fix my previous problem but the new error comes up

```
Feat extract conv layer 0 was initialized from feature_extractor.conv_layers.0.0.weight.
Traceback (most recent call last):
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 162, in <module>
    convert_wav2vec2_checkpoint(args.checkpoint_path, args.pytorch_dump_folder_path, args.dict_path)
  File "/usr/local/lib/python3.6/dist-packages/torch/autograd/grad_mode.py", line 26, in decorate_context
    return func(*args, **kwargs)
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 151, in convert_wav2vec2_checkpoint
    recursively_load_weights(model, hf_wav2vec)
  File "convert.py", line 77, in recursively_load_weights
    hf_model.config.feat_extract_norm == "group",
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 112, in load_conv_layer
    value.shape == feature_extractor.conv_layers[layer_id].conv.bias.data.shape
AttributeError: 'NoneType' object has no attribute 'data'
```

> ```
> AttributeError: 'NoneType' object has no attribute 'data'
> ```

Hey @acul3, 

thanks for trying out the `XLSR` model! I haven't added it to the conversion script yet - will do it this week!

thank you for the great idea and work to merge wav2vec2 to transformers. I am wondering:
1. how to use a transformer LM for decoding as Fairseq uses wav2letter's decoder for better accuracy. 
2. it seems to be much convenient if the output has a confidence score too

> thank you for the great idea and work to merge wav2vec2 to transformers. I am wondering:
> 
> 1. how to use a transformer LM for decoding as Fairseq uses wav2letter's decoder for better accuracy.
> 2. it seems to be much convenient if the output has a confidence score too

1. I'm also still working on figuring out the best way to do this!
2. Yeah that will be a nice-to-have, but it will require some time to be added.

@patrickvonplaten Another great PR! 
I am wondering whether this current implementation supports self-supervise training of user's custom dataset ?


> supervise

Not yet :-) Working on it right now!

Inspiring feat Patrick! I remember this model was like a puzzle for me the first time I tried to make it work. You've made it incredibly easy to use. Can't wait for the decoders and finetuning


> > ```
> > AttributeError: 'NoneType' object has no attribute 'data'
> > ```
> 
> Hey @acul3,
> 
> thanks for trying out the `XLSR` model! I haven't added it to the conversion script yet - will do it this week!

hi @patrickvonplaten is this available yet? thank you

@patrickvonplaten 
Strange bug, if I use the self trained lv60 960h version on CPU the results are very good
Using it on cuda the results are pretty strange
I am using the code provided on model card

> > ```
> > AttributeError: 'NoneType' object has no attribute 'data'
> > ```
> 
> 
> Hey @acul3,
> 
> 
> thanks for trying out the `XLSR` model! I haven't added it to the conversion script yet - will do it this week!

I am trying to convert `XLSR` models too, by modifying the config below, it seems that I used all weights as wav2vec_small do
```
{
  "activation_dropout": 0.0,
  "apply_spec_augment": true,
  "architectures": [
    "Wav2Vec2Model"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 1,
  "conv_bias": true,
  "conv_dim": [
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "conv_kernel": [
    10,
    3,
    3,
    3,
    3,
    2,
    2
  ],
  "conv_stride": [
    5,
    2,
    2,
    2,
    2,
    2,
    2
  ],
  "do_stable_layer_norm": false,
  "eos_token_id": 2,
  "feat_extract_activation": "gelu",
  "feat_extract_norm": "layer",
  "feat_proj_dropout": 0.1,
  "final_dropout": 0.0,
  "freeze_feat_extract_train": true,
  "hidden_act": "gelu",
  "hidden_dropout": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "gradient_checkpointing": true,
  "layer_norm_eps": 1e-05,
  "layerdrop": 0.0,
  "mask_channel_length": 10,
  "mask_channel_min_space": 1,
  "mask_channel_other": 0.0,
  "mask_channel_prob": 0.0,
  "mask_channel_selection": "static",
  "mask_time_length": 10,
  "mask_time_min_space": 1,
  "mask_time_other": 0.0,
  "mask_time_prob": 0.05,
  "mask_time_selection": "static",
  "model_type": "wav2vec2",
  "no_mask_channel_overlap": false,
  "no_mask_time_overlap": false,
  "num_attention_heads": 16,
  "num_conv_pos_embedding_groups": 16,
  "num_conv_pos_embeddings": 128,
  "num_feat_extract_layers": 7,
  "num_hidden_layers": 24,
  "pad_token_id": 0
}
```
After converting:
> Unused weights ['quantizer.vars', 'quantizer.weight_proj.weight', 'quantizer.weight_proj.bias', 'project_q.weight', 'project_q.bias', 'layer_norm.weight', 'layer_norm.bias', 'final_proj.weight', 'final_proj.bias']

Nevertheless, The logits are the same on testing, seems that I still left something not converted? Do you have any ideas what's going on ? 
```
tokenizer = Wav2Vec2Tokenizer.from_pretrained("facebook/wav2vec2-base")
model = Wav2Vec2Model.from_pretrained("./xlsr")

def map_to_array(batch):
  speech, _ = sf.read(batch["file"])
  batch["speech"] = speech
  return batch

ds = load_dataset("patrickvonplaten/librispeech_asr_dummy", "clean", split="validation")
ds = ds.map(map_to_array)

input_values = tokenizer(ds["speech"][0], return_tensors="pt").input_values  # Batch size 1
logits = model(input_values).last_hidden_state
```

Logits:
```
tensor([[[ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684],
         [ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684],
         [ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684],
         ...,
         [ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684],
         [ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684],
         [ 0.2285,  0.6376,  0.0164,  ...,  0.0275,  0.1215, -0.2684]]],
       grad_fn=<NativeLayerNormBackward>)
```