Error Multi-Node Training with Deepspeed
Author: ShivamSharma2705
Pull request: no

### System Info

- `transformers` version: 4.24.0.dev0
- Platform: Linux-3.10.0-1127.el7.x86_64-x86_64-with-glibc2.17
- Python version: 3.8.13
- Huggingface_hub version: 0.10.1
- PyTorch version (GPU?): 1.8.2+cu111 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: <fill in>
- Using distributed or parallel set-up in script?: <fill in>

### Who can help?

@stas00 

### Information

- [X] The official example scripts
- [ ] My own modified scripts

### Tasks

- [X] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

This error occurred when I was running run_clm.py script with deepspeed.

1. Configure the hostfile provided to deepspeed as follows

         node1 slots=1
         node2 slots=2


### Expected behavior

The run gets stuck at the following for around 30mins for the node I am running the scripts from

node1: [2023-01-30 08:34:30,454] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in D$
epSpeed with backend nccl

After 30 mins or so, I get the following error

node2: Traceback (most recent call last):
node2:   File "run_clm.py", line 659, in <module>
node2:     main()
node2:   File "run_clm.py", line 244, in main
node2:     model_args, data_args, training_args = parser.parse_args_into_dataclasses()
node2:   File "/home/transformers/src/transformers/hf_argparser.py", lin$
 226, in parse_args_into_dataclasses
node2:     obj = dtype(**inputs)
node2:   File "<string>", line 104, in __init__
node2:   File "/home/transformers/src/transformers/training_args.py", li$
e 1118, in __post_init__
node2:     and (self.device.type != "cuda")
node2:   File "/home/transformers/src/transformers/utils/import_utils.py$
, line 1000, in wrapper
node2:     return func(*args, **kwargs)
node2:   File "/home/transformers/src/transformers/training_args.py", li$
e 1478, in device
node2:     return self._setup_devices
node2:   File "/home/transformers/src/transformers/utils/generic.py", li$
e 57, in __get__
node2:     cached = self.fget(obj)
node2:   File "/home/transformers/src/transformers/utils/import_utils.py$
, line 1000, in wrapper
node2:     return func(*args, **kwargs)
node2:   File "/home/transformers/src/transformers/training_args.py", li$
e 1413, in _setup_devices
node2:     deepspeed.init_distributed()
node2:   File "/home/anaconda3/envs/deepspeed_hf/lib/python3.8/site-packages/deepspeed/comm/$
omm.py", line 637, in init_distributed
node2:     cdb = TorchBackend(dist_backend, timeout, init_method)
node2:   File "/home/anaconda3/envs/deepspeed_hf/lib/python3.8/site-packages/deepspeed/comm/$
orch.py", line 30, in __init__
node2:     self.init_process_group(backend, timeout, init_method)
node2:   File "/home/anaconda3/envs/deepspeed_hf/lib/python3.8/site-packages/deepspeed/comm/$
orch.py", line 34, in init_process_group
node2:     torch.distributed.init_process_group(backend,
node2:   File "/home/anaconda3/envs/deepspeed_hf/lib/python3.8/site-packages/torch/distribut$
d/distributed_c10d.py", line 500, in init_process_group
node2:     store, rank, world_size = next(rendezvous_iterator)
node2:   File "/home/anaconda3/envs/deepspeed_hf/lib/python3.8/site-packages/torch/distribut$
d/rendezvous.py", line 190, in _env_rendezvous_handler
node2:     store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
node2: RuntimeError: connect() timed out.

Comments:
30min is the timeout for any NCCL operations.

I assume it never started training? Was it doing anything during the 30min, like pre-processing a dataset?

please run again and do
```
pip install py-spy
py-spy dump --pid PID 
```
PID of the process that is stuck.

And of course your Issue doesn't tell us anything about how to reproduce it or even which program you had a problem with.

The code was stuck at this

node1: [2023-01-30 08:34:30,454] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in D$
epSpeed with backend nccl

The output for the py-spy command for node 1 is 
Thread 39975 (idle): "MainThread"
    _try_wait (subprocess.py:1764)
    _wait (subprocess.py:1806)
    wait (subprocess.py:1083)
    main (deepspeed/launcher/runner.py:522)
    <module> (deepspeed:6)

To reproduce just run [run_clm.py](https://github.com/huggingface/transformers/blob/main/examples/pytorch/language-modeling/run_clm.py) with deepspeed and in the hostfile add two nodes as follows

node1 slots=1
node2 slots=2

thank you, yes, so you're not launching deepspeed properly. 

what is `node1`? it has to be an actual hostname, what is `slots=1`

what is the actual setup - you have 2 nodes with 2 gpus each? then it'd be something like:

```
hostname1 slots=2
hostname2 slots=2
```



Yeah, two nodes with 1 GPU each

hostname1 slots=1
hostname2 slots=1


most likely you then have an ssh issue where it gets stuck in trying to connect to those nodes.

As we have cleared out that this is not an integration issue - please reopen this question at https://github.com/microsoft/DeepSpeed/issues 

and when you report it probably start with a simple test script so it's easier to reproduce / isolate to the deepspeed launcher. e.g. you can use this script https://github.com/stas00/toolbox/blob/master/pytorch/torch-distributed-gpu-test.py but you will need to adapt your launching to the way you're trying to do (the instructions inside the script don't use `deepspeed` as the launcher).