question: wrong global rank when trying multi-nodes
context: Hello,
  I'm trying to train my model with multi-nodes (2 nodes, 8 gpus per each, using ddp accelator & trying without using slurm)
  But I got problem with GLOBAL_RANK
  in node 1,
  initializing ddp: GLOBAL_RANK: 1, MEMBER: 2/16
  ...
  initializing ddp: GLOBAL_RANK: 7, MEMBER: 8/16
  
  same as in node 2,
  initializing ddp: GLOBAL_RANK: 0, MEMBER: 1/16
  ...
  initializing ddp: GLOBAL_RANK: 7, MEMBER: 8/16
  
  And got stuck with repeated message like below
  Waiting in store based barrier to initialize process group for rank: 3, key: store_based_barrier_key:1 (world_size=16, worker_count=13, timeout=0:30:00)
  Waiting in store based barrier to initialize process group for rank: 1, key: store_based_barrier_key:1 (world_size=16, worker_count=13, timeout=0:30:00)
  
  I'm trying to setup like this document but also got problem, like below
   os.environ["MASTER_ADDR"] = master_addr
   os.environ["MASTER_PORT"] = master_port
   os.environ["WORLD_SIZE"] = "16"
   os.environ["NODE_RANK"] = rank
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py", line 474, in fit
      self.accelerator.setup(self, model)
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/gpu.py", line 19, in setup
      return super().setup(trainer, model)
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 69, in setup
      self.connect_training_type_plugin(self.training_type_plugin, model)
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/accelerators/accelerator.py", line 328, in connect_training_type_plugin
      plugin.connect(model)
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/parallel.py", line 68, in connect
      self.setup(model)
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py", line 95, in setup
      self.task_idx = self.cluster_environment.local_rank()
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/site-packages/pytorch_lightning/plugins/environments/torchelastic_environment.py", line 48, in local_rank
      return int(os.environ['LOCAL_RANK'])
    File "/home/user/.pyenv/versions/3.7.9/lib/python3.7/os.py", line 681, in __getitem__
      raise KeyError(key) from None
  
  I'd appreciate any help. thanks in advance
answer: What have you set for MASTER_ADDR and MASTER_PORT? These have to reference to one of the two machines you are using. For example if I have two nodes like this:
  IP: 512.124.134.4
  IP: 512.124.136.8
  And I want 512.124.134.4 to be my master node.
  For both my machines I'd need to run something like MASTER_ADDR=512.124.134.4 MASTER_PORT=4500 python train.py.
  Let me know if this helps! On top of this, we should update the doc if this does work :)