Multi GPU Whisper Training Error!
Author: xyx361100238
Pull request: no

### System Info

- `transformers` version: 4.28.0.dev0
- Platform: Linux-5.4.0-165-generic-x86_64-with-debian-bullseye-sid
- Python version: 3.7.16
- Huggingface_hub version: 0.13.2
- PyTorch version (GPU?): 1.13.1+cu116 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@sanchit-gandhi @pacman100 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

use officially script(changed loaddata) to finetune my own datasets, it works in single GPU, according to [multi GPU train](https://github.com/huggingface/transformers/tree/main/examples/pytorch/speech-recognition#multi-gpu-whisper-training)：Attempting to use torch.distributed.launch / torch.distributed.run / torchrun methods, will result in an error message stating that the log file has been uploaded
[run.log](https://github.com/huggingface/transformers/files/13069037/run.log)


### Expected behavior

The step of ‘preprocess train dataset’ is no longer limited to 30 minutes and can complete multi card concurrent training

Comments:
param：
CUDA_VISIBLE_DEVICES=0,1 torchrun --nproc_per_node=2 run_speech_recognition_seq2seq.py \
	--model_name_or_path=${modFile} \
	--data_list=${listPath} \
	--do_train \
	--language=${lang} \
	--task="transcribe" \
	--output_dir=${outPath} \
	--per_device_train_batch_size="16" \
	--per_device_eval_batch_size="16" \
	--auto_find_batch_size="True" \
	--logging_steps="5" \
	--learning_rate="1e-5" \
	--warmup_steps="500" \
	--num_train_epochs=50 \
	--evaluation_strategy="steps" \
	--eval_steps="5000" \
	--save_strategy="steps" \
	--save_steps="5000" \
	--save_total_limit=10 \
	--load_best_model_at_end="True" \
	--generation_max_length="225" \
	--preprocessing_num_workers="32" \
	--length_column_name="input_length" \
	--max_duration_in_seconds="30" \
	--audio_column_name="audio" \
	--text_column_name="sentence" \
	--freeze_feature_encoder="False" \
	--gradient_checkpointing \
	--fp16 \
	--group_by_length \
	--cache_dir="/nasStore/000-CacheData/transfomer_whisper_map_cache/" \
	--overwrite_output_dir \
	--predict_with_generate \
	--optim="adamw_torch"

The failure is occurring in a multi-GPU set-up since there is a time-out limit of 30 minutes when running multiple processes in PyTorch.

For large datasets being used for multi-GPU training it is advised to run the pre-processing ahead of time on a **single machine** by appending the `--preprocessing_only` flag to your command:
https://github.com/huggingface/transformers/blob/d2a980ec74db8bc3ea0104e126cbf5d4b1f0e73b/examples/pytorch/speech-recognition/run_speech_recognition_seq2seq.py#L512

You can then remove the flag, and launch with a multi-GPU set-up as you have done before. The dataset will be loaded straight from cache, so pre-processing won't be performed a second time.

Otherwise, if you want all GPUs to be waiting while you pre-process the dataset, you can use your original command, but with the `--ddp_timeout` flag set to the number of seconds for the multi-GPU timeout in seconds. You can set it to something large like 10800 (3 hours). But this is slightly wasteful since we don't need to use any GPUs for the pre-processing!

thanks！I've already done this according to document