Wav2vec2 Pretraining issue
Author: Kshitizkhandel
Pull request: no

### System Info

```shell
transformers version-4.11.3
```


### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

https://colab.research.google.com/drive/1kepA7ryMG7YmNtSYjiJjBM984KRbpZuV#scrollTo=LdIxS2EEgMmz

### Expected behavior

```shell
I tried to run the pre training demo of wav2vec2 on libri speech but i run into this error
of unrecognised arguments:/ or no module found 'transformers.modeling_outputs
```


### Checklist

- [X] I have read the migration guide in the readme. ([pytorch-transformers](https://github.com/huggingface/transformers#migrating-from-pytorch-transformers-to-transformers); [pytorch-pretrained-bert](https://github.com/huggingface/transformers#migrating-from-pytorch-pretrained-bert-to-transformers))
- [X] I checked if a related official extension example runs on my machine.

Comments:
The script you are using probably requires a more recent version of Transformers. cc @sanchit-gandhi 

As @sgugger has mentioned, you can relax the pinning constraints on all your HF libraries and install them from their latest versions:
```
!pip install transformers datasets accelerate jiwer
```

Also, you've got to be careful downgrading `torchaudio` like that as it can mess-up your torch installation. My recommendation would be to leave the default `torchaudio` version and update the Unix package `ffmpeg` to version 4:

```
!add-apt-repository -y ppa:jonathonf/ffmpeg-4
!apt update
!apt install -y ffmpeg 
```

(taken from this blog https://huggingface.co/blog/fine-tune-whisper#prepare-environment)


Hi @Kshitizkhandel :) - I can run it on `4.25.0.dev0` so it is a problem with a version.

> As @sgugger has mentioned, you can relax the pinning constraints on all your HF libraries and install them from their latest versions:
> 
> ```
> !pip install transformers datasets accelerate jiwer
> ```
> 
> Also, you've got to be careful downgrading `torchaudio` like that as it can mess-up your torch installation. My recommendation would be to leave the default `torchaudio` version and update the Unix package `ffmpeg` to version 4:
> 
> ```
> !add-apt-repository -y ppa:jonathonf/ffmpeg-4
> !apt update
> !apt install -y ffmpeg 
> ```
> 
> (taken from this blog https://huggingface.co/blog/fine-tune-whisper#prepare-environment)



> As @sgugger has mentioned, you can relax the pinning constraints on all your HF libraries and install them from their latest versions:
> 
> ```
> !pip install transformers datasets accelerate jiwer
> ```
> 
> Also, you've got to be careful downgrading `torchaudio` like that as it can mess-up your torch installation. My recommendation would be to leave the default `torchaudio` version and update the Unix package `ffmpeg` to version 4:
> 
> ```
> !add-apt-repository -y ppa:jonathonf/ffmpeg-4
> !apt update
> !apt install -y ffmpeg 
> ```
> 
> (taken from this blog https://huggingface.co/blog/fine-tune-whisper#prepare-environment)

Thank you for your swift response on this. @sanchit-gandhi I'm running out of disk space even on google colab pro, do you recommend using colab pro plus/ or anything that you suggest?

Hey @Kshitizkhandel. The full LibriSpeech dataset contains approx 1000h of labelled audio data. As such, it requires ~130GB disk space to download and prepare, which is why you're running into trouble on a Google Colab! Pre-training is an intensive task, both for compute and storage, so it's only really advised if you really need to do it. Otherwise, you can take a pre-trained checkpoint and fine-tune it for your downstream application (see https://huggingface.co/blog/fine-tune-wav2vec2-english).

May I ask what the purpose is for pre-training? Are you simply wanting to try it for yourself, or do you have the intent of pre-training on your own custom dataset?

The reason that I ask is that the 'official' Wav2Vec2 checkpoints are pre-trained on 60,000h of audio data. These checkpoints are available to you on the HF Hub:
- https://huggingface.co/facebook/wav2vec2-base
- https://huggingface.co/facebook/wav2vec2-large-lv60

-> so if you want a pre-trained checkpoint, you can skip the pre-training and load any of these pre-trained checkpoints

There are also multiple fine-tuned checkpoints that you can use out-of-the-box for downstream speech recognition:
- https://huggingface.co/facebook/wav2vec2-base-960h
- https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self

-> see the aforementioned blog for fine-tuning these checkpoints

Closing this issue as the original question has been solved! Feel free to ask on the forum if you require any additional assistance with your task and I'll be more than happy to help @Kshitizkhandel! https://discuss.huggingface.co 

Best of luck!

> Hey @Kshitizkhandel. The full LibriSpeech dataset contains approx 1000h of labelled audio data. As such, it requires ~130GB disk space to download and prepare, which is why you're running into trouble on a Google Colab! Pre-training is an intensive task, both for compute and storage, so it's only really advised if you really need to do it. Otherwise, you can take a pre-trained checkpoint and fine-tune it for your downstream application (see https://huggingface.co/blog/fine-tune-wav2vec2-english).
> 
> May I ask what the purpose is for pre-training? Are you simply wanting to try it for yourself, or do you have the intent of pre-training on your own custom dataset?
> 
> The reason that I ask is that the 'official' Wav2Vec2 checkpoints are pre-trained on 60,000h of audio data. These checkpoints are available to you on the HF Hub:
> 
> * https://huggingface.co/facebook/wav2vec2-base
> * https://huggingface.co/facebook/wav2vec2-large-lv60
> 
> -> so if you want a pre-trained checkpoint, you can skip the pre-training and load any of these pre-trained checkpoints
> 
> There are also multiple fine-tuned checkpoints that you can use out-of-the-box for downstream speech recognition:
> 
> * https://huggingface.co/facebook/wav2vec2-base-960h
> * https://huggingface.co/facebook/wav2vec2-large-960h-lv60-self
> 
> -> see the aforementioned blog for fine-tuning these checkpoints
> 
> Closing this issue as the original question has been solved! Feel free to ask on the forum if you require any additional assistance with your task and I'll be more than happy to help @Kshitizkhandel! https://discuss.huggingface.co
> 
> Best of luck!

I've fine-tuned projects and was looking to try something new but given the hassles it doesn't seem to be a conducive option. Thanks for your help on this @sanchit-gandhi 

Pre-training is certainly possible, you just need a lot of disk space and compute time for it to be worthwhile!

If you want to try something new, you can check out the Whisper model from OpenAI 😉 https://huggingface.co/blog/fine-tune-whisper This gets very good results with very little fine-tuning!

> Pre-training is certainly possible, you just need a lot of disk space and compute time for it to be worthwhile!
> 
> If you want to try something new, you can check out the Whisper model from OpenAI 😉 https://huggingface.co/blog/fine-tune-whisper This gets very good results with very little fine-tuning!

Yeah, I read your lucid and well articulated blog on it. Great work!