Problem generating pytorch_model.bin for wav2vec2 large model using convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py
Author: mzboito
Pull request: no

## Environment info
- `transformers` version: 4.17.0
- Platform: Linux-5.14.0-1031-oem-x86_64-with-glibc2.29
- Python version: 3.8.10
- PyTorch version (GPU?): 1.11.0+cu102 (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: No
- Using distributed or parallel set-up in script?: No


### Who can help
@patrickvonplaten, @anton-l

## Information

I'm uploading new wav2vec2 models to the LeBenchmark (https://huggingface.co/LeBenchmark). 
We already have several models for which the integration with HuggingFace works great. 
Example: https://huggingface.co/LeBenchmark/wav2vec2-FR-7K-large
 
**The issue:** I am not able to generate the _pytorch_model.bin_ files for my new *large* models using the same _config.json_ files that worked fine a few months ago. The script (transformers/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py) is giving me a size mismatch error for the _quantizer.vars_. I have no problem generating the same .bin (with a different _config.json_ file) for my new *base* models.

It seems to me that this script is incorrectly retrieving the shape of some layers inside the wav2vec2, and I don't know how to fix it.

## To reproduce
For generating the pytorch_model.bin model for my new large models, I'm running the following command on my local machine:

```python3 transformers/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py --pytorch_dump_folder_path . --checkpoint_path <checkpoint_best.pt> --not_finetuned --config_path <config.json>```

An example of checkpoint_best.pt and its corresponding config.json can be found here: https://huggingface.co/LeBenchmark/wav2vec-FR-1K-Female-large/tree/main

My error is the following:
```
Traceback (most recent call last):
  File "/home/mzanonboito/IS_2022/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 263, in <module>
    convert_wav2vec2_checkpoint(
  File "/home/mzanonboito/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/home/mzanonboito/IS_2022/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 248, in convert_wav2vec2_checkpoint
    recursively_load_weights(model, hf_wav2vec, not is_finetuned)
  File "/home/mzanonboito/IS_2022/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 133, in recursively_load_weights
    set_recursively(hf_model, mapped_key, value, name, weight_type)
  File "/home/mzanonboito/IS_2022/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 79, in set_recursively
    raise ValueError(
ValueError: Shape of hf  is torch.Size([1, 640, 192]), but should be torch.Size([1, 640, 384]) for quantizer.vars
```
At config.json we can find in line 9 the correct shape for this variable:
```
"codevector_dim": 384,
```
Finally, loading the model directly on  torch, we can verify that it is indeed the correct shape.
```
>>> import torch
>>> model = torch.load("checkpoint_best.pt")
>>> model['model']['quantizer.vars'].size()
torch.Size([1, 640, 384])
```
I even tried changed codevector_dim at config.json to 192 to see if the script would then work. The result is a new shape error:
```
File "/home/mzanonboito/IS_2022/transformers/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 79, in set_recursively
    raise ValueError(
ValueError: Shape of hf  is torch.Size([1, 640, 96]), but should be torch.Size([1, 640, 384]) for quantizer.vars
```

I thus don't understand where this issue is coming from, and I would appreciate your help. Thanks!

## Expected behavior

For base models, this script call works fine, and the output files can be found here:
https://huggingface.co/LeBenchmark/wav2vec-FR-1K-Female-base/tree/main


Comments:
Hey @mzboito,

Thanks for the very in-detail issue. I think the problem is the following:

Since Wav2Vec2 using product quantization (quite often with `num_groups=2`), the last dimension of the tensor should be ``codevector_dim // 2 (=num_groups)`` see here: https://github.com/huggingface/transformers/blob/c65633156b29482256619ee6515ba3d6c24578aa/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L974

This should be equivalent to fairseq here: https://github.com/pytorch/fairseq/blob/8fce12ddd4a0414de5a726af6193cee4893f0a13/fairseq/modules/gumbel_vector_quantizer.py#L52

This means in order for the conversion to function, we should set `codevector_dim = 768`. 
I think there was one other problem in the config - I took the liberty to correct this in your repo here: https://huggingface.co/LeBenchmark/wav2vec-FR-1K-Female-large/commit/fadfcc59c210ef559b08d8e16b6289278acedc1b and added the converted HF checkpoint. Hope that's fine for you :-)


Thanks a lot for the help and quick answer!!!! :-)