How to use spawn to start sub-process

Ardeal: <p>Hi,</p>
<p>I would like to start 2 processes on my computer with 2 GPUs. spawn function is used to start 2 processes.</p>
<p>Question 1:<br>
how to specify rank number for each process when I use spawn function to start main_worker?</p>
<p>Question 2:<br>
how to specify/check local_rank of each process in main_worker?</p>
<p>Question 3:<br>
world_size means total number of GPUs used in the processes.<br>
rank is the index of each processes/process number.<br>
local_rank means the GPU index used in the process(rank).<br>
is my understanding correct?</p>
<p>My test code is pasted as follows.</p>
<pre><code class="lang-python">import torch
import torch.distributed as dist
import torch.multiprocessing as mp

mp.spawn(main_worker, nprocs=2, args=(2, myargs))

def main_worker(proc, nprocs, args):

   dist.init_process_group(backend='nccl', init_method='tcp://127.0.0.1:23456', world_size=2, rank=gpu)
   torch.cuda.set_device(args.local_rank)

   train_dataset = ...
   train_sampler = torch.utils.data.distributed.DistributedSampler(train_dataset)

   train_loader = torch.utils.data.DataLoader(train_dataset, batch_size=..., sampler=train_sampler)

   model = ...
   model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[args.local_rank])

   optimizer = optim.SGD(model.parameters())

   for epoch in range(100):
      for batch_idx, (data, target) in enumerate(train_loader):
          images = images.cuda(non_blocking=True)
          target = target.cuda(non_blocking=True)
          ...
          output = model(images)
          loss = criterion(output, target)
          ...
          optimizer.zero_grad()
          loss.backward()
          optimizer.step()

</code></pre>

rvarm1: <aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="126190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>how to specify rank number for each process when I use spawn function to start main_worker?</p>
</blockquote>
</aside>
<p>The method you start with <code>mp.spawn</code> must take in as its first argument a <code>rank</code> parameter (<code>proc</code>) in your example, which will be the rank of the process. Ranks are assigned in order of the processes starting in each worker.</p>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="126190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>how to specify/check local_rank of each process in main_worker?</p>
</blockquote>
</aside>
<p>After initializing the process group (as you have with <code>init_process_group</code>), you can use the <code>dist.get_rank()</code> API to get the global rank of the process in the world. To get the local rank, assuming a homogenous set up, mod the result of <code>dist.get_rank()</code> by the number of GPUs on the machine.</p>
<aside class="quote no-group" data-username="Ardeal" data-post="1" data-topic="126190">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ardeal/48/33257_2.png" class="avatar"> Ardeal:</div>
<blockquote>
<p>Question 3:<br>
world_size means total number of GPUs used in the processes.<br>
rank is the index of each processes/process number.<br>
local_rank means the GPU index used in the process(rank).<br>
is my understanding correct?</p>
</blockquote>
</aside>
<p>Your understanding of <code>rank</code> and <code>local_rank</code> is correct, though note that <code>local_rank</code> is not necessarily the GPU index. You can assign GPUs to ranks non-sequentially using <code>torch.cuda.set_device()</code> API, for example rank 0 could operate on GPU 1 and rank 1 could operate on GPU 0. Regarding world-size, world_size = total no. of active processes across all nodes. Usually this will be equal to the no. of GPUs available</p>

Ardeal: <p><a class="mention" href="/u/rvarm1">@rvarm1</a> ,<br>
Many thanks for your reply!</p>
<p>one more question:<br>
what is global_rank?  and how to get/set global_rank in code?</p>

Ardeal: <p><a class="mention" href="/u/rvarm1">@rvarm1</a> ,</p>
<p>I am trying to run torch code on vscode on Ubuntu. The code is running on Ubuntu.<br>
I am experiencing the following issue:</p>
<pre><code class="lang-python">Exception has occurred: ProcessRaisedException


-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/torch/multiprocessing/spawn.py", line 59, in _wrap
    fn(i, *args)
  File "/home/smb/code_python/pytorch-image-models/train.py", line 523, in main
    torch.distributed.init_process_group(backend='nccl', init_method='env://')
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/distributed_c10d.py", line 520, in init_process_group
    store, rank, world_size = next(rendezvous_iterator)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/rendezvous.py", line 199, in _env_rendezvous_handler
    store = TCPStore(master_addr, master_port, world_size, start_daemon, timeout)
RuntimeError: Address already in use
  File "/home/smb/code_python/pytorch-image-models/train.py", line 1089, in &lt;module&gt;
    mp.spawn(main, nprocs=2, args=(2, [333,444,555]))
</code></pre>
<p>I set the env like this:</p>
<pre><code class="lang-python">            "env": {
                "CUDA_VISIBLE_DEVICES": "6,7",
                "WORLD_SIZE": "2",
                "RANK": "0",
                "MASTER_ADDR": "127.0.0.1",
                "MASTER_PORT": "44147",
            }
</code></pre>
<pre><code class="lang-python">mp.spawn(main, nprocs=2, args=(2, [333,444,555]))

args = parser.parse_args()

def main(process_index, process_cnt:int, args_spawn:list=[]):

	args.device = 'cuda:%d' % args.local_rank
	torch.cuda.set_device(args.local_rank)
	torch.distributed.init_process_group(backend='nccl', init_method='env://')
	args.world_size = torch.distributed.get_world_size()
	args.rank = torch.distributed.get_rank()

</code></pre>