Enable Wav2Vec2 Pretraining
Author: patrickvonplaten
Pull request: no
Labels: name: Good First Issue

name: Good Second Issue

# 🚀 Feature request

This is a feature request to add Wav2Vec2 Pretraining functionality to the transformers library. This is a "Good Second Issue" feature request, which means that interested contributors should have some experience with the transformers library and ideally also with training/fine-tuning Wav2Vec2.

## Motivation

The popular [Wav2Vec2](https://huggingface.co/models?filter=wav2vec2) model cannot be pretrained using the Hugging Face library yet. During the fine-tuning week, multiple people have reported improved results by pretraining wav2vec2 directly on the target language before fine-tuning it.

## Your contribution

I am happy to give an interesting contributor guidance throughout the PR and answer all relevant questions.

## How to start

1) To begin with, one should run a pretraining forward pass using the official Wav2Vec2 repository. The forward pass can be found here: https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L474. 
It is important that the argument `features_only` is set to `False` in the [`forward`](https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L474) function.
Succesfully running a forward pass with fairseq is important to ensure the correctness of the hugging face implementation by comparing the two outputs.

This is probably the most difficult part of the PR.

**Note:** this also means that the loaded fairseq wav2vec2 checkpoint should include weights for the `GumbelVectorQuantizer` quantizer, see: https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L277

The easiest checkpoint to try out pretraining with is probably the wav2vec2 2.0 Base - No fine-tuning [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt)

[Here](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#train-a-wav2vec-20-base-model) is the official Fairseq recipe on how to do so.

2) Having run a forward pass successfully, the methods can now be implemented into transformers [here](https://github.com/huggingface/transformers/blob/653076ca307520ee85fd5f5de6918019f8521bb5/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L966) as a new class that could roughly look as follows:

```python 

class Wav2Vec2ForPretraining:

def init(self, config):
      self.wav2vec2 = Wav2Vec2Model(config)
      self.quantizer = ...
      self.project_q = ...

def forward(...):

        outputs = self.wav2vec2(
            input_values,
            attention_mask=attention_mask,
            output_attentions=output_attentions,
            output_hidden_states=output_hidden_states,
            return_dict=return_dict,
        )

        # ... all the pretraining logic here

```

Having implemented the class it should be made sure that a forward pass of `Wav2Vec2ForPretraining` works.

3) Convert the pretrained checkpoints correctly

After `Wav2Vec2ForPretraining` was succesfully added, a "non-finuted" checkpoint, e.g., the wav2vec2 2.0 Base - No fine-tuning [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt) should be converted to the hugging face models. One will probably have to slightly adapt the conversion script as well: https://github.com/huggingface/transformers/blob/master/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py

Having converted the checkpoint it can be uploaded to the hub and checked whether the checkpoint yields the same outputs as the official wav2vec2 pretraining functionality.

4) Add tests

Next, a couple of tests should be implemented that make sure that the behavior stays correct in the future. This included both fast and "slow" integration tests (fast tests are "normal" tests); "slow" integration tests include a "real" checkpoint and test its output against a hardcoded expected output tensor slice as it's done, *e.g.*, [here](https://github.com/huggingface/transformers/blob/653076ca307520ee85fd5f5de6918019f8521bb5/tests/test_modeling_big_bird.py#L823).

## Ask for help

For questions regarding how to finish 1), they can be directly asked on this issue; I will try to answer them as best as I can. Also gently pinging @cceyda here as I think she has already succesfully pretrained a wav2vec2 model using fairseq. Hope it's fine to ping you here 😅) - in case you have some good tips on how to pretrained wav2vec2 with fairseq, it would be amazing to share some tips here. 

For questions when doing 2), 3) & 4) please directly asked on the PR you have opened to implement the model.



Comments: