text: 
  
  TITLE
  Enable Wav2Vec2 Pretraining
  
  COMMENTS
  0
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 15
  rocket: 0
  eyes: 0
  
  BODY
  # 🚀 Feature request
  
  This is a feature request to add Wav2Vec2 Pretraining functionality to the transformers library. This is a "Good Second Issue" feature request, which means that interested contributors should have some experience with the transformers library and ideally also with training/fine-tuning Wav2Vec2.
  
  ## Motivation
  
  The popular [Wav2Vec2](https://huggingface.co/models?filter=wav2vec2) model cannot be pretrained using the Hugging Face library yet. During the fine-tuning week, multiple people have reported improved results by pretraining wav2vec2 directly on the target language before fine-tuning it.
  
  ## Your contribution
  
  I am happy to give an interesting contributor guidance throughout the PR and answer all relevant questions.
  
  ## How to start
  
  1) To begin with, one should run a pretraining forward pass using the official Wav2Vec2 repository. The forward pass can be found here: https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L474. 
  It is important that the argument `features_only` is set to `False` in the [`forward`](https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L474) function.
  Succesfully running a forward pass with fairseq is important to ensure the correctness of the hugging face implementation by comparing the two outputs.
  
  This is probably the most difficult part of the PR.
  
  **Note:** this also means that the loaded fairseq wav2vec2 checkpoint should include weights for the `GumbelVectorQuantizer` quantizer, see: https://github.com/pytorch/fairseq/blob/436166a00c2ecd1215df258f022608947cca2aa8/fairseq/models/wav2vec/wav2vec2.py#L277
  
  The easiest checkpoint to try out pretraining with is probably the wav2vec2 2.0 Base - No fine-tuning [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt)
  
  [Here](https://github.com/pytorch/fairseq/tree/master/examples/wav2vec#train-a-wav2vec-20-base-model) is the official Fairseq recipe on how to do so.
  
  2) Having run a forward pass successfully, the methods can now be implemented into transformers [here](https://github.com/huggingface/transformers/blob/653076ca307520ee85fd5f5de6918019f8521bb5/src/transformers/models/wav2vec2/modeling_wav2vec2.py#L966) as a new class that could roughly look as follows:
  
  ```python 
  
  class Wav2Vec2ForPretraining:
  
  def init(self, config):
        self.wav2vec2 = Wav2Vec2Model(config)
        self.quantizer = ...
        self.project_q = ...
  
  def forward(...):
  
          outputs = self.wav2vec2(
              input_values,
              attention_mask=attention_mask,
              output_attentions=output_attentions,
              output_hidden_states=output_hidden_states,
              return_dict=return_dict,
          )
  
          # ... all the pretraining logic here
  
  ```
  
  Having implemented the class it should be made sure that a forward pass of `Wav2Vec2ForPretraining` works.
  
  3) Convert the pretrained checkpoints correctly
  
  After `Wav2Vec2ForPretraining` was succesfully added, a "non-finuted" checkpoint, e.g., the wav2vec2 2.0 Base - No fine-tuning [here](https://dl.fbaipublicfiles.com/fairseq/wav2vec/wav2vec_small.pt) should be converted to the hugging face models. One will probably have to slightly adapt the conversion script as well: https://github.com/huggingface/transformers/blob/master/src/transformers/models/wav2vec2/convert_wav2vec2_original_pytorch_checkpoint_to_pytorch.py
  
  Having converted the checkpoint it can be uploaded to the hub and checked whether the checkpoint yields the same outputs as the official wav2vec2 pretraining functionality.
  
  4) Add tests
  
  Next, a couple of tests should be implemented that make sure that the behavior stays correct in the future. This included both fast and "slow" integration tests (fast tests are "normal" tests); "slow" integration tests include a "real" checkpoint and test its output against a hardcoded expected output tensor slice as it's done, *e.g.*, [here](https://github.com/huggingface/transformers/blob/653076ca307520ee85fd5f5de6918019f8521bb5/tests/test_modeling_big_bird.py#L823).
  
  ## Ask for help
  
  For questions regarding how to finish 1), they can be directly asked on this issue; I will try to answer them as best as I can. Also gently pinging @cceyda here as I think she has already succesfully pretrained a wav2vec2 model using fairseq. Hope it's fine to ping you here 😅) - in case you have some good tips on how to pretrained wav2vec2 with fairseq, it would be amazing to share some tips here. 
  
  For questions when doing 2), 3) & 4) please directly asked on the PR you have opened to implement the model.
  
  