Add new meta w2v2-conformer BERT-like model
Author: ylacombe
Pull request: yes

# What does this PR do?

Meta just open-sourced a Wav2Vec2-BERT conformer [model](https://huggingface.co/facebook/w2v-bert-2.0). This one is particularly interesting because it's under a MIT license and was pretrained on 101 input languages!
 
It requires adaption to the current w2v2-conformer code, which this PR does.

cc @sanchit-gandhi, @Vaibhavs10 and @amyeroberts 

Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_28165). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thanks for the many good suggestions @sanchit-gandhi, could you take one last look? 

Also cc @amyeroberts, could you review this PR? Normally I'd wait for @sanchit-gandhi's approval before asking your opinion, but as the end-of-year vacations are approaching, I'd rather speed up the process! 

Hi @amyeroberts, happy new year!

I've addressed your feedback! 
Two last remarks:
1. [this commit](https://github.com/huggingface/transformers/pull/28165/commits/4848a26c1d7c8e03ba225767b26a541d3054dead) is there to facilitate training
2. this new checkpoint doesn't use `Wav2Vec2Processor` but a fbank feature extractor, here `SeamlessM4TFeatureExtractor`. Consequence -> the input name is `input_features` instead of `input_values`. Do you think we should leave this as it is ?

Let me know if you need other changes !

I'm opening a discussion after an offline discussion with @sanchit-gandhi on the relevance of adding a totally new modeling code instead of modifying the one of Wav2Vec2Conformer.

Main reason behind it, this new model uses a totally different feature extractor with two main consequences:
1. the input name is `input_features` instead of `input_values`
2. it totally gets rid of the first block of Wav2Vec2Conformer.

There also differences in architecture that could justify this choice.

I have no personal position on the subject as both options (leaving this PR as it is or adding a new model) have pros and cons. The only thing is that if we choose to keep this PR, I'd have to add `input_features` as a possible input of the different Wav2Vec2Conformer models.

WDYT @amyeroberts ?

+1 on @ArthurZucker's comment - @ylacombe Yes, a separate modeling file I think makes sense. 

Hey @ArthurZucker and @amyeroberts, thanks for your feedback. I've updated the codebase to add a new model instead of modifying the existing one! 
Could you review this PR again ?

Note that one failing test (torch&jax) is unrelated to my change (probably flaky) and the other one relies on a trained model checkpoint, and there are none available yet -> I'm training one at the moment and will use it if it is ready at the moment of merging. Otherwise, I'll remove/update the related docstrings 

Hey @sanchit-gandhi, thanks for the review here ! Will iterate soon!

Just a QQ regarding the pre-training task, you're right, it differs with the fairseq2 implementation so I'll remove the corresponding class for now. However, it raises the question of how to convert the weights, which was for now done with the pretraining class. My opinion is to do it with whatever other main class (e.g the CTC one) and being clear that the CTC head is not trained in the docs and the model card. WDYT ?



Could you not convert it with `Wav2Vec2BERTModel`? It's the same as the pre-training class but without the codebook weights?

Also one point of discussion: should the model name be camel-cased as is now the convention in Transformers (`Wav2Vec2BertModel`)? This is a bit of an edge case since BERT was original named as `BERTModel` (upper-case), so it's a question of whether we want consistency with BERT, or want to adhere to the camel-case conventions.

> Could you not convert it with Wav2Vec2BERTModel? It's the same as the pre-training class but without the codebook weights?

Every downstream model (`Wav2Vec2BERTForPreTraining`, `Wav2Vec2BERTForCTC`, `Wav2Vec2BERTForSequenceClassification`, etc.) actually has `Wav2Vec2BertModel` as a parameter (`self.wav2vec2_bert`). 

So I could convert the weights to `Wav2Vec2BERTModel`, but actually using `CLASS.from_pretrained(REPO_ID)` won't work, right ?


> Also one point of discussion: should the model name be camel-cased as is now the convention in Transformers (Wav2Vec2BertModel)? This is a bit of an edge case since BERT was original named as BERTModel (upper-case), so it's a question of whether we want consistency with BERT, or want to adhere to the camel-case conventions.

I'll let @ArthurZucker or @amyeroberts decide on this one !

This should be handled by `base_model_prefix`:
```python
from transformers import Wav2Vec2BERTConfig, Wav2Vec2BERTModel, Wav2Vec2BERTForCTC
import torch

config = Wav2Vec2BERTConfig()
random_model = Wav2Vec2BERTModel(config)

random_model.save_pretrained("./output_dir")
loaded_model = Wav2Vec2BERTForCTC.from_pretrained("./output_dir", vocab_size=32)

# check a layer-norm layer is loaded correctly
print(torch.allclose(loaded_model.wav2vec2_bert.feature_projection.layer_norm.weight, random_model.feature_projection.layer_norm.weight))
```
**Output:**
```
Some weights of Wav2Vec2BERTForCTC were not initialized from the model checkpoint at ./output_dir 
and are newly initialized: ['lm_head.bias', 'lm_head.weight']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions 
and inference.
True
```
=> we see that all the encoder weights are loaded correctly, and just the LM head newly initialised

Hey @amyeroberts, thanks for the review ! I've addressed all your comments, let me know if that works with you !