Keep waiting for push command to finish at the end of running run_speech_recognition_ctc.py 
Author: IvanLauLinTiong
Pull request: no

## Environment info

- `transformers` version: 4.17.0.dev0
- Platform: Linux-5.11.0-37-generic-x86_64-with-glibc2.10 ( running at OVH Cloud 's AI Training Jobs)
- Python version: 3.8.8
- PyTorch version (GPU?): 1.10.2+cu102 (True)
- Tensorflow version (GPU?): not installed
- Using GPU in script?: Yes
- Using distributed or parallel set-up in script?: No

### Who can help
@patrickvonplaten  @anton-l 

Models:
 facebook/wav2vec2-xls-r-300m

Library:
- Speech: @patrickvonplaten, @anton-l

## Information

Model I am using facebook/wav2vec2-xls-r-300m:

The problem arises when using:
* [ ] the official example scripts: https://github.com/huggingface/transformers/blob/master/examples/pytorch/speech-recognition/run_speech_recognition_ctc.py


The tasks I am working on is:
* [ ] dataset:  mozilla-foundation/common_voice_8_0

## Problem Description
I ran training script `run_speech_recognition_ctc.py`  in 100 epochs. After finishing training and evaluation at last step, the program yet shutting down and keeps waiting for the push command to finish: 
![image](https://user-images.githubusercontent.com/23013350/152737186-81602a1c-9c54-4324-9a00-16c44291aa3f.png)
 
However, everything is pushed to hub successfully. So, at the end, I `Ctrl + C` to quit the program manually.

## To reproduce
I ran the official training script `run_speech_recognition_ctc.py` and the command is shown below:

`python run_speech_recognition_ctc.py \
	--dataset_name="mozilla-foundation/common_voice_8_0" \
	--model_name_or_path="facebook/wav2vec2-xls-r-300m" \
	--dataset_config_name="zh-HK" \
	--output_dir="./" \
	--num_train_epochs="100" \
	--per_device_train_batch_size="32" \
    --per_device_eval_batch_size="16" \
	--gradient_accumulation_steps="2" \
	--learning_rate="3e-4" \
	--warmup_steps="500" \
	--text_column_name="sentence" \
	--length_column_name="input_length" \
	--layerdrop="0.0" \
	--save_total_limit="3" \
	--freeze_feature_encoder \
	--gradient_checkpointing \
	--fp16 \
	--group_by_length \
    --use_auth_token \
	--push_to_hub \
	--do_train \
    --do_eval \
	--max_duration_in_seconds="6" \
	--evaluation_strategy='epoch' \
	--save_strategy='epoch' \`

Notes:  You may change `--num_train_epochs="100"` to shorter runn as the behaviour still persist.


## Expected behavior
The program itself should shut down gracefully after finishing the training, evaluation and pushing to hub steps.


Comments:
Thanks for opening this issue! I've actually never such an issue before - @sgugger any idea what the problem could be here? 

Not really. It's coming from `huggingface_hub` async push, so deferring to @LysandreJik :-)

Hey @IvanLauLinTiong, have you verified the push has finished, and the weights were updated on the hub? It should tell you that as long as the push is unfinished. If the program were to terminate early, then the push would be interrupted.

Hi @LysandreJik , 

May I know what is the last default commit message for the weights file if uploaded properly to the hub?  I just ran the script `run_speech_recognition_ctc.py` with everything default in 100 epochs, and my `pytorch_model.bin` latest commit message is `Training in progress, epoch 100`. What is the default commit message if it is uploaded successfully?

Can you help me to check?
Here is my repo link: https://huggingface.co/ivanlau/wav2vec2-large-xls-r-300m-cantonese/tree/main



That's because some people also encountered this issue during the Robust Speech Event (screenshot below) and  prolly I was too fast to assume this is a HF bug. 

![image](https://user-images.githubusercontent.com/23013350/152911424-05353b61-ddc7-4838-8979-afde242e1791.png)


![image](https://user-images.githubusercontent.com/23013350/152911475-9ec4f967-f71d-4d5f-84af-065d0dd023b0.png)



If this is not a bug, hmmm, shouldn't the warning message be more descriptive a bit ? like 'currently pushing file A....' or some progress bar. :)

Thanks.

Thanks for bringing it up! I'm not certain it's a bug from one of our libraries, I'd say it's probably linked to a very slow upload speed. I agree that the message could be better.

However, you shouldn't be afraid of losing anything: the files are committed, so you can feel free to Ctrl+C, and check in the local folder to see if the files have been pushed. You can do so by doing the `git status` command, which should tell you if your local clone is up to date with the remote, or if some commits have not been pushed.

If some commits have not been pushed, simply do `git push` :)

Thanks for your issue!

@LysandreJik  
ok thanks 👍 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.