Run audio classification example using "facebook/hubert-base-ls960" model got stuck when use deepspeed, but works for wav2vec2
Author: Hertin
Pull request: no

### System Info

I was trying to run the audio classification example using "facebook/hubert-base-ls960" with deepspeed on a 3 gpu node. The trainer and the model got stuck at the first training step. However, if I change only the  `--model_name_or_path=facebook/hubert-base-ls960`  to `--model_name_or_path=facebook/wav2vec2-base`, I am able to run the audio classification example using "facebook/wav2vec2-base" without problem. Not sure if other people also have this issue.


### Who can help?

_No response_

### Information

- [X] The official example scripts
- [X] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

1. clone and install huggingface repo `https://github.com/huggingface/transformers.git`
2. go to `examples/pytorch/audio-classification` folder
3. save the config into config/stage2.json
```
{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    },
    "optimizer": {
        "type": "AdamW",
        "params": {
            "lr": "auto",
            "weight_decay": "auto",
            "torch_adam": true,
            "adam_w_mode": true
        }
    },
    "scheduler": {
        "type": "WarmupDecayLR",
        "params": {
            "warmup_min_lr": "auto",
            "warmup_max_lr": "auto",
            "warmup_num_steps": "auto",
            "total_num_steps": "auto"
        }
    },
    "zero_optimization": {
        "stage": 2,
        "allgather_partitions": true,
        "allgather_bucket_size": 2e8,
        "overlap_comm": true,
        "reduce_scatter": true,
        "reduce_bucket_size": "auto",
        "contiguous_gradients": true
    },
    "gradient_accumulation_steps": 1,
    "gradient_clipping": "auto",
    "steps_per_print": 2000,
    "train_batch_size": "auto",
    "train_micro_batch_size_per_gpu": "auto",
    "wall_clock_breakdown": false
}
```
4. execute the following command line and the scripts stucks the the first step of training
```
deepspeed --num_gpus=3 run_audio_classification.py \
    --model_name_or_path facebook/hubert-base-ls960 \
    --dataset_name common_language \
    --audio_column_name audio \
    --label_column_name language \
    --output_dir ac-base-lang-id \
    --overwrite_output_dir \
    --remove_unused_columns False \
    --do_train \
    --do_eval \
    --fp16 \
    --learning_rate 3e-4 \
    --max_length_seconds 16 \
    --attention_mask True \
    --warmup_ratio 0.1 \
    --num_train_epochs 10 \
    --per_device_train_batch_size 8 \
    --gradient_accumulation_steps 1 \
    --per_device_eval_batch_size 1 \
    --dataloader_num_workers 8 \
    --logging_strategy steps \
    --logging_steps 10 \
    --evaluation_strategy epoch \
    --save_strategy epoch \
    --load_best_model_at_end True \
    --metric_for_best_model accuracy \
    --save_total_limit 3 \
    --seed 0 \
    --cache_dir /nws/user/hertin/.cache/huggingface \
    --deepspeed config/stage2.json
```


### Expected behavior

5. the tail of the output looks like this and the training got stuck:
```No modifications detected for re-loaded extension module utils, skipping build step...
Loading extension module utils...
Time to load utils op: 0.0006883144378662109 seconds
[INFO|trainer.py:1777] 2023-06-08 15:09:13,470 >> ***** Running training *****
[INFO|trainer.py:1778] 2023-06-08 15:09:13,470 >>   Num examples = 22,194
[INFO|trainer.py:1779] 2023-06-08 15:09:13,470 >>   Num Epochs = 10
[INFO|trainer.py:1780] 2023-06-08 15:09:13,470 >>   Instantaneous batch size per device = 8
[INFO|trainer.py:1781] 2023-06-08 15:09:13,470 >>   Total train batch size (w. parallel, distributed & accumulation) = 24
[INFO|trainer.py:1782] 2023-06-08 15:09:13,470 >>   Gradient Accumulation steps = 1
[INFO|trainer.py:1783] 2023-06-08 15:09:13,470 >>   Total optimization steps = 9,250
[INFO|trainer.py:1784] 2023-06-08 15:09:13,471 >>   Number of trainable parameters = 90,379,693
  0%|                                                                                                                       | 0/9250 [00:00<?, ?it/s]
```

Comments:
cc @sanchit-gandhi @pacman100 

That's super weird - is there any activity on your GPU? Thought it might be an issue with SpecAug + DeepSpeed, but we have the config params set the same in HuBERT and Wav2Vec2. So it must be a DeepSpeed bug. Are you able to see with any more detail where it hands? E.g. maybe with a torch profile, or just killing the programme when it hangs and seeing which line it got to?

When it gets stuck, the GPU utility goes to 100%. When I Ctrl+C to interrupt, I see the following traceback:
```
 0%|                                                                                 | 0/9250 [00:00<?, ?it/s]

C[2023-06-12 10:59:44,481] [INFO] [launch.py:314:sigkill_handler] Killing subprocess 1234292
2023-06-12 10:59:44,581] [INFO] [launch.py:314:sigkill_handler] Killing subprocess 1234292
raceback (most recent call last):
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/bin/deepspeed", line 6, in <module>
   main()
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/lib/python3.9/site-packages/deepspeed/launcher/runner
py", line 570, in main
   result.wait()
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/lib/python3.9/subprocess.py", line 1189, in wait
   return self._wait(timeout=timeout)
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/lib/python3.9/subprocess.py", line 1917, in _wait
   (pid, sts) = self._try_wait(0)
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/lib/python3.9/subprocess.py", line 1875, in _try_wait
   (pid, sts) = os.waitpid(self.pid, wait_flags)
 File "/nws/user/hertin/softwares/miniconda3/envs/slullm/lib/python3.9/site-packages/deepspeed/launcher/runner
py", line 562, in sigkill_handler
   result_kill = subprocess.Popen(kill_cmd, env=env)
ameError: free variable 'kill_cmd' referenced before assignment in enclosing scope
2023-06-12 10:59:44,867] [INFO] [launch.py:314:sigkill_handler] Killing subprocess 1234293
 [2023-06-12 10:59:45,152] [INFO] [launch.py:314:sigkill_handler] Killing subprocess 1234294
2023-06-12 10:59:45,515] [INFO] [launch.py:323:sigkill_handler] Main process received SIGTERM, exiting
```

Hey @Hertin, this indeed looks like a DeepSpeed bug (i.e. we see deepspeed hang before launching the process). Not sure why we're getting it for HuBERT and not Wav2Vec2 though 🤔 Could you verify your transformers version + deepspeed version please?
```
transformers-cli env
python -c "import deepspeed; print(deepspeed.__version__)"
```

Thanks for the reply. This is my transformer and deepspeed version
```
transformers-cli env
```
```
Setting ds_accelerator to cuda (auto detect)

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.31.0.dev0
- Platform: Linux-5.4.0-144-generic-x86_64-with-glibc2.31
- Python version: 3.9.16
- Huggingface_hub version: 0.15.1
- Safetensors version: 0.3.1
- PyTorch version (GPU?): 2.0.1+cu117 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>
```

```
python -c "import deepspeed; print(deepspeed.__version__)"
```
```
Setting ds_accelerator to cuda (auto detect)
0.9.3
```

We can try installing to the latest version of deepspeed (0.9.4) but I don't think this is going to fix it... I'm not sure here - I think this is a question for the deepspeed repo to be honest! If you could share with them a reproducible code snippet of the issue they should be able to dive deeper on why deepspeed is hanging in the way it is. Unfortunately this is out of the scope of transformers at this point

Thanks for your time. I will close this issue as it is more likely a deepspeed issue.

Thanks @Hertin and sorry we were not able to find a fix! Hope it goes well asking on the DS repo