Fine-tune wave2vec trained checkpoint
Author: Omarnabk
Pull request: no

The tutorial of wav2vec says that the encoder model was originally trained on the common voice dataset. but all the fine-tuning tasks were trained on datasets from common voice. When fine-tuning wav2vec2 model, we fine-tuning the final linear classification layer only, right? 

If this is the case, why don't you release the lm.weights and lm.biese for each language? 

I am asking because I have a fine-tuned wav2vec2 model and it is doing very well. I want to fine-tune only the last liner layer, without initializing with random values. How can I load the lm.weights and lm.biese from a checkpoint? 


I tried to load the pre-trained checkpoint like this 
```
import torch 
state_dict = torch.load('./eng_asr/pytorch_model.bin', map_location='cpu')
```

Then initialize a new pre-trained model

```
import torch 
model = Wav2Vec2ForCTC.from_pretrained(f'facebook/wav2vec2-large-xlsr-53', 
                                       vocab_size=processor.tokenizer.vocab_size, 
                                      pad_token_id=processor.tokenizer.pad_token_id)
model.freeze_feature_extractor()
model.load_state_dict(state_dict)

```

At this stage, I'm loading the weights of my checkpoint to facebook/wav2vec2-large-xlsr-53. 

Then, I set the training arguments

```
training_args = TrainingArguments(  
    output_dir=repo_name,
    group_by_length=True,
    length_column_name = 'input_length',
    per_device_train_batch_size=2,
    per_device_eval_batch_size=2,
    gradient_accumulation_steps=1,
    evaluation_strategy="steps",
    num_train_epochs=10,
    fp16=True,
    save_steps=500,
    eval_steps=500,
    logging_steps=500,
    learning_rate=5e-4,
    warmup_steps=500,
    save_total_limit=4,
)

```

then, when I initialize Trainer, it takes like 3G of the GPU memory 

```
trainer = Trainer(
    model=model,
    data_collator=data_collator,
    args=training_args,
    compute_metrics=compute_metrics,
    train_dataset=train_data,
    eval_dataset=test_data,
    tokenizer=processor.feature_extractor
)

```

When I hit train, regardless of the batch size, I get out of memory. I have 24G GPU RAM. I don't understand why the memory gone in one second!

Comments:
cc @anton-l 

Hi @Omarnabk! The model is actually pretrained with an unsupervised objective (it never saw the target transcriptions) which is why there are no official pretrained LM heads available :slightly_smiling_face: 

As far as I can see, you're doing everything correctly, but the audio clips in your dataset might be too long for the model to process on a 24G GPU. Try chunking your data into sentences, or filtering speech samples that are larger than about 20sec.

Let me know if I understood your case correctly :slightly_smiling_face: 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.