convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py works for data2vec 1.0 checkpoint but not data2vec 2.0
Author: alanrice
Pull request: no

### System Info

- `transformers` version: 4.21.3
- Platform: Linux-5.4.0-122-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.12.0
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): 2.9.2 (True)
- Flax version (CPU?/GPU?/TPU?): 0.6.3 (gpu)
- Jax version: 0.4.1
- JaxLib version: 0.4.1

### Who can help?

@sanchit-gandhi @sgugger

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

- Download [data2vec 1.0 Large (No fine-tuning) .pt](https://dl.fbaipublicfiles.com/fairseq/data2vec/vox_pretrained.pt) from [fairseq/data2vec](https://github.com/facebookresearch/fairseq/tree/main/examples/data2vec) 
- Download config.json from [facebook/data2vec-audio-large](https://huggingface.co/facebook/data2vec-audio-large/blob/main/config.json)
- Run [convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/data2vec/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py)
- pytorch_model.bin output

```shell
python scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py \
--pytorch_dump_folder_path converted \
--checkpoint_path vox_pretrained.pt \
--config_path config.json \
--not_finetuned
``` 
<details><summary>Output</summary>
<p>
2023-04-29 15:56:57 | INFO | fairseq.tasks.text_to_speech | Please install tensorboardX: pip install tensorboardX
loading configuration file config.json
Model config Data2VecAudioConfig {
  "_name": "data2vec_audio",
  "activation_dropout": 0.1,
  "adapter_kernel_size": 3,
  "adapter_stride": 2,
  "add_adapter": false,
  "apply_spec_augment": true,
  "architectures": [
    "Data2VecAudioModel"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 1,
  "classifier_proj_size": 256,
  "codevector_dim": 768,
  "contrastive_logits_temperature": 0.1,
  "conv_bias": false,
  "conv_dim": [
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "conv_kernel": [
    10,
    3,
    3,
    3,
    3,
    2,
    2
  ],
  "conv_pos_kernel_size": 19,
  "conv_stride": [
    5,
    2,
    2,
    2,
    2,
    2,
    2
  ],
  "ctc_loss_reduction": "sum",
  "ctc_zero_infinity": false,
  "diversity_loss_weight": 0.1,
  "do_stable_layer_norm": true,
  "eos_token_id": 2,
  "feat_extract_activation": "gelu",
  "feat_extract_dropout": 0.0,
  "feat_extract_norm": "layer",
  "feat_proj_dropout": 0.1,
  "feat_quantizer_dropout": 0.0,
  "final_dropout": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout": 0.1,
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "layerdrop": 0.0,
  "mask_feature_length": 10,
  "mask_feature_min_masks": 0,
  "mask_feature_prob": 0.0,
  "mask_time_length": 10,
  "mask_time_min_masks": 2,
  "mask_time_prob": 0.05,
  "model_type": "data2vec-audio",
  "num_adapter_layers": 3,
  "num_attention_heads": 16,
  "num_codevector_groups": 2,
  "num_codevectors_per_group": 320,
  "num_conv_pos_embedding_groups": 16,
  "num_conv_pos_embeddings": 5,
  "num_feat_extract_layers": 7,
  "num_hidden_layers": 24,
  "num_negatives": 100,
  "output_hidden_size": 1024,
  "pad_token_id": 0,
  "proj_codevector_dim": 768,
  "tdnn_dilation": [
    1,
    2,
    3,
    1,
    1
  ],
  "tdnn_dim": [
    512,
    512,
    512,
    512,
    1500
  ],
  "tdnn_kernel": [
    5,
    3,
    3,
    1,
    1
  ],
  "torch_dtype": "float32",
  "transformers_version": "4.21.3",
  "use_weighted_layer_sum": false,
  "vocab_size": 32,
  "xvector_output_dim": 512
}

2023-04-29 15:58:39 | WARNING | datasets.builder | Reusing dataset librispeech_asr_dummy (/root/.cache/huggingface/datasets/patrickvonplaten___librispeech_asr_dummy/clean/2.1.0/f2c70a4d03ab4410954901bde48c54b85ca1b7f9bf7d616e7e2a72b5ee6ddbfc)
It is strongly recommended to pass the ``sampling_rate`` argument to this function. Failing to do so can result in silent errors that might be hard to debug.
torch.Size([4, 666, 1024]) torch.Size([4, 666, 1024])
max_absolute_diff = 8.707307279109955e-05
Do both models output the same tensors? 🔥
Configuration saved in converted/config.json
Model weights saved in converted/pytorch_model.bin
Feature extractor saved in converted/preprocessor_config.json
</p>
</details> 

- Download [data2vec 2.0 Large (No fine-tuning) .pt](https://dl.fbaipublicfiles.com/fairseq/data2vec2/large_vox.pt) from [fairseq/data2vec](https://github.com/facebookresearch/fairseq/tree/main/examples/data2vec)
- Download config.json from [facebook/data2vec-audio-large](https://huggingface.co/facebook/data2vec-audio-large/blob/main/config.json)
- Run [convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/data2vec/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py)
- KeyError: 'final_proj.0.weight'

```shell
python scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py \
--pytorch_dump_folder_path converted \
--checkpoint_path large_vox.pt \
--config_path config.json \
--not_finetuned
``` 
<details><summary>Output</summary>
<p>
2023-04-29 15:59:58 | INFO | fairseq.tasks.text_to_speech | Please install tensorboardX: pip install tensorboardX
loading configuration file config.json
Model config Data2VecAudioConfig {
  "_name": "data2vec_audio",
  "activation_dropout": 0.1,
  "adapter_kernel_size": 3,
  "adapter_stride": 2,
  "add_adapter": false,
  "apply_spec_augment": true,
  "architectures": [
    "Data2VecAudioModel"
  ],
  "attention_dropout": 0.1,
  "bos_token_id": 1,
  "classifier_proj_size": 256,
  "codevector_dim": 768,
  "contrastive_logits_temperature": 0.1,
  "conv_bias": false,
  "conv_dim": [
    512,
    512,
    512,
    512,
    512,
    512,
    512
  ],
  "conv_kernel": [
    10,
    3,
    3,
    3,
    3,
    2,
    2
  ],
  "conv_pos_kernel_size": 19,
  "conv_stride": [
    5,
    2,
    2,
    2,
    2,
    2,
    2
  ],
  "ctc_loss_reduction": "sum",
  "ctc_zero_infinity": false,
  "diversity_loss_weight": 0.1,
  "do_stable_layer_norm": true,
  "eos_token_id": 2,
  "feat_extract_activation": "gelu",
  "feat_extract_dropout": 0.0,
  "feat_extract_norm": "layer",
  "feat_proj_dropout": 0.1,
  "feat_quantizer_dropout": 0.0,
  "final_dropout": 0.1,
  "gradient_checkpointing": false,
  "hidden_act": "gelu",
  "hidden_dropout": 0.1,
  "hidden_dropout_prob": 0.1,
  "hidden_size": 1024,
  "initializer_range": 0.02,
  "intermediate_size": 4096,
  "layer_norm_eps": 1e-05,
  "layerdrop": 0.0,
  "mask_feature_length": 10,
  "mask_feature_min_masks": 0,
  "mask_feature_prob": 0.0,
  "mask_time_length": 10,
  "mask_time_min_masks": 2,
  "mask_time_prob": 0.05,
  "model_type": "data2vec-audio",
  "num_adapter_layers": 3,
  "num_attention_heads": 16,
  "num_codevector_groups": 2,
  "num_codevectors_per_group": 320,
  "num_conv_pos_embedding_groups": 16,
  "num_conv_pos_embeddings": 5,
  "num_feat_extract_layers": 7,
  "num_hidden_layers": 24,
  "num_negatives": 100,
  "output_hidden_size": 1024,
  "pad_token_id": 0,
  "proj_codevector_dim": 768,
  "tdnn_dilation": [
    1,
    2,
    3,
    1,
    1
  ],
  "tdnn_dim": [
    512,
    512,
    512,
    512,
    1500
  ],
  "tdnn_kernel": [
    5,
    3,
    3,
    1,
    1
  ],
  "torch_dtype": "float32",
  "transformers_version": "4.21.3",
  "use_weighted_layer_sum": false,
  "vocab_size": 32,
  "xvector_output_dim": 512
}

Traceback (most recent call last):
  File "/notebooks/scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py", line 287, in <module>
    convert_wav2vec2_checkpoint(
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/notebooks/scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py", line 213, in convert_wav2vec2_checkpoint
    state_dict["model"]["final_proj.weight"] = state_dict["model"].pop("final_proj.0.weight")
KeyError: 'final_proj.0.weight'
</p>
</details>

### Expected behavior

Expected behavior is that model weights are saved in pytorch_model.bin for both data2vec 1.0 and 2.0 checkpoints.

Comments:
Getting beyond the key error by either commenting [lines 212-213](https://github.com/huggingface/transformers/blob/849367ccf741d8c58aa88ccfe1d52d8636eaf2b7/src/transformers/models/data2vec/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py#L212C50-L213 ) or changing the keys to `modality_encoders.AUDIO.decoder.proj.weight` and `modality_encoders.AUDIO.decoder.proj.bias`, results in a `Could not infer model type from {cfg}` error.

```
Traceback (most recent call last):
  File "/notebooks/scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py", line 287, in <module>
    convert_wav2vec2_checkpoint(
  File "/usr/local/lib/python3.9/dist-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "/notebooks/scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py", line 227, in convert_wav2vec2_checkpoint
    model = load_data2vec(converted_ckpt)
  File "/notebooks/scripts/convert_data2vec_audio_original_pytorch_checkpoint_to_pytorch.py", line 224, in load_data2vec
    model, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([path])
  File "/notebooks/fairseq/fairseq/checkpoint_utils.py", line 484, in load_model_ensemble_and_task
    model = task.build_model(cfg.model, from_checkpoint=True)
  File "/notebooks/fairseq/fairseq/tasks/audio_pretraining.py", line 178, in build_model
    model = super().build_model(model_cfg, from_checkpoint)
  File "/notebooks/fairseq/fairseq/tasks/fairseq_task.py", line 355, in build_model
    model = models.build_model(cfg, self, from_checkpoint)
  File "/notebooks/fairseq/fairseq/models/__init__.py", line 101, in build_model
    f"Could not infer model type from {cfg}. "
KeyError: "'_name'"
``` 

It looks like the architecture of data2vec 2.0 is different from 1.0, so supporting this would require changing the modeling code for Data2Vec in Transformers or adding a new Data2Vec2 model. Patching the existing conversion script likely won't be sufficient.

Please note that the conversion script are provided as an indication from the model contributor on how they converted the original checkpoint to the Hugging Face format. They are not maintained and not expected to work on other checkpoints.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.

Leaving this as closed since the issue requires a new conversion script and new modelling code for data2vec2 (rather than being an issue with the existing data2vec code). Feel free to open a feature request if this is something you'd like to see @alanrice!