wrong global rank when trying multi-nodes
Hello,
I'm trying to train my model with multi-nodes (2 nodes, 8 gpus per each, using ddp accelator & trying without using slurm)
But I got problem with GLOBAL_RANK
in node 1,
initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/16
...
initializing ddp: GLOBAL_RANK: 7, MEMBER: 8/16

same as in node 2,
initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/16
...
initializing ddp: GLOBAL_RANK: 7, MEMBER: 8/16

And got stuck with repeated message like below
Waiting in store based barrier to initialize process group for rank: 3, key: store_based_barrier_key:1 (world_size=16, worker_count=13, timeout=0:30:00)
Waiting in store based barrier to initialize process group for rank: 1, key: store_based_barrier_key:1 (world_size=16, worker_count=13, timeout=0:30:00)

I'm trying to setup like this document but also got problem, like below
 os.environ["MASTER_ADDR"] = master_addr
 os.environ["MASTER_PORT"] = master_port
 os.environ["WORLD_SIZE"] = "16"
 os.environ["NODE_RANK"] = rank
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 474, in fit
    self.accelerator.setup(self, model)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu.py", line 19, in setup
    return super().setup(trainer, model)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 69, in setup
    self.connect_training_type_plugin(self.training_type_plugin, model)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 328, in connect_training_type_plugin
    plugin.connect(model)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/parallel.py", line 68, in connect
    self.setup(model)
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 95, in setup
    self.task_idx = self.cluster_environment.local_rank()
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/environments/torchelastic_environment.py", line 48, in local_rank
    return int(os.environ['LOCAL_RANK'])
  File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/os.py", line 681, in __getitem__
    raise KeyError(key) from None

I'd appreciate any help. thanks in advance
What have you set for MASTER_ADDR and MASTER_PORT? These have to reference to one of the two machines you are using. For example if I have two nodes like this:
IP: 512.124.134.4
IP: 512.124.136.8
And I want 512.124.134.4 to be my master node.
For both my machines I'd need to run something like MASTER_ADDR=512.124.134.4 MASTER_PORT=4500 python train.py.
Let me know if this helps! On top of this, we should update the doc if this does work :)