convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py does not work on fairseq wav2vec2-xls-r weights
Author: heatz123
Pull request: no

### System Info

- `transformers` version: 4.22.2
- Platform: Linux-5.4.0-126-generic-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.10.0
- PyTorch version (GPU?): 1.12.1+cu102 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no

### Who can help?

@patrickvonplaten

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

on bash,
1. wget https://dl.fbaipublicfiles.com/fairseq/wav2vec/xlsr2_300m.pt
2. wget https://huggingface.co/facebook/wav2vec2-xls-r-300m/raw/main/config.json
3. python3 convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py --pytorch_dump_folder_path output --checkpoint_path xlsr2_300m.pt --config_path config.json --not_finetuned

Then this error arises:
```
Traceback (most recent call last):
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 273, in <module>
    convert_wav2vec2_checkpoint(
  File "/home/heatz123/env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context
    return func(*args, **kwargs)
  File "convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py", line 254, in convert_wav2vec2_checkpoint
    model, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([checkpoint_path])
  File "/home/heatz123/env/lib/python3.8/site-packages/fairseq/checkpoint_utils.py", line 436, in load_model_ensemble_and_task
    task = tasks.setup_task(cfg.task)
  File "/home/heatz123/env/lib/python3.8/site-packages/fairseq/tasks/__init__.py", line 39, in setup_task
    cfg = merge_with_parent(dc(), cfg)
  File "/home/heatz123/env/lib/python3.8/site-packages/fairseq/dataclass/utils.py", line 500, in merge_with_parent
    merged_cfg = OmegaConf.merge(dc, cfg)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/omegaconf.py", line 321, in merge
    target.merge_with(*others[1:])
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 331, in merge_with
    self._format_and_raise(key=None, value=None, cause=e)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 329, in merge_with
    self._merge_with(*others)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 347, in _merge_with
    BaseContainer._map_merge(self, other)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/basecontainer.py", line 314, in _map_merge
    dest[key] = src._get_node(key)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/dictconfig.py", line 258, in __setitem__
    self._format_and_raise(
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/base.py", line 95, in _format_and_raise
    format_and_raise(
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/_utils.py", line 629, in format_and_raise
    _raise(ex, cause)
  File "/home/heatz123/env/lib/python3.8/site-packages/omegaconf/_utils.py", line 610, in _raise
    raise ex  # set end OC_CAUSE=1 for full backtrace
omegaconf.errors.ConfigKeyError: Key 'multiple_train_files' not in 'AudioPretrainingConfig'
        full_key: multiple_train_files
        reference_type=Optional[AudioPretrainingConfig]
        object_type=AudioPretrainingConfig
```

note that I am using fairseq 0.12.2 (which was installed by default using `pip install fairseq`)

### Expected behavior

converting fairseq -> pytorch(transformers) weights should work without any errors.

Comments:
And after some inspections on fairseq library, I found that this change can make the conversion work also on wav2vec2-xls-r weights:
https://github.com/huggingface/transformers/blob/bc21aaca789f1a366c05e8b5e111632944886393/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py#L249

to
```python
        task_arg = argparse.Namespace(task='audio_pretraining')
        task = fairseq.tasks.setup_task(task_arg)
        model, _, _ = fairseq.checkpoint_utils.load_model_ensemble_and_task([checkpoint_path], task=task)
```

Can I make a PR on this change?

Yes please! Would you like to open a PR for this change? You can also update the conversion script for Wav2Vec2 Conformer - it's exactly the same logic in `convert_wav2vec2_conformer_checkpoint` 🤗

I see, thank you. Will open a PR within a day.