Using Pytorch's multiprocessing along with distributed package

Usama-Zafar: <p>I am trying to spawn a couple of process using pytorch’s multiprocessing module within a openmpi distributed back-end. What I have is the following code:</p>
<pre><code>def run(rank_local, rank, world_size, maingp):
    print("I WAS SPAWNED ", rank_local, " OF ", rank)
    
    tensor = torch.zeros(1)
    tensor += 1
    
    if rank == 0:
        tensor += 100
        dist.send(tensor, dst=1)
    else:
        print("I am spawn: ", rank, "and my tensor value before receive: ", tensor[0])
        dist.recv(tensor, src=0)
        print("I am spawn: ", rank, "and my tensor value after  receive: ", tensor[0])
    

if __name__ == '__main__':
    
    # Initialize Process Group
    dist.init_process_group(backend="mpi", group_name="main")
    maingp = None #torch.distributed.new_group([0,1])
    mp.set_start_method('spawn')    
    
    # get current process information
    world_size = dist.get_world_size()
    rank = dist.get_rank()
    
    # Establish Local Rank and set device on this node
    mp.spawn(run, args=(rank, world_size, maingp), nprocs=1)
</code></pre>
<p>I run this code using the openmpi as follows:</p>
<pre><code>mpirun -n 2 python code.py
</code></pre>
<p>So my understanding is that mpirun creates two process with ranks [0, 1], each of these process spawn new process with their local rank as 0. Now if I want to communicate between these two sub-processes of the main process I get some Traceback and following error:</p>
<pre><code>-- Process 0 terminated with the following error:
Traceback (most recent call last):
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/multiprocessing/spawn.py", line 19, in _wrap
    fn(i, *args)
  File "/home/usama/code/test/code.py", line 19, in run
    dist.send(tensor, dst=1)
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 666, in send
    _check_default_pg()
  File "/home/usama/anaconda3/lib/python3.6/site-packages/torch/distributed/distributed_c10d.py", line 191, in _check_default_pg
    "Default process group is not initialized"
AssertionError: Default process group is not initialized
</code></pre>
<p>My question is how do I make these sub-processes to be able to communicate i.e the [0, 0] process sending something to [1, 0] process. Any ideas?</p>
<p>I have asked this on stack-overflow as well but to no avail!!!<br>
<aside class="onebox stackexchange">
  <header class="source">
      <a href="https://stackoverflow.com/questions/56054164/using-pytorchs-multiprocessing-along-with-distributed" target="_blank" rel="nofollow noopener">stackoverflow.com</a>
  </header>
  <article class="onebox-body">
      <a href="https://stackoverflow.com/users/2555668/usama-zafar" target="_blank" rel="nofollow noopener">
    <img alt="Usama Zafar" src="https://i.stack.imgur.com/048Fx.jpg?s=128&amp;g=1" class="thumbnail onebox-avatar" width="128" height="128">
  </a>
<h4>
  <a href="https://stackoverflow.com/questions/56054164/using-pytorchs-multiprocessing-along-with-distributed" target="_blank" rel="nofollow noopener">Using Pytorch's Multiprocessing along with Distributed</a>
</h4>

<div class="tags">
  <strong>python, pytorch, openmpi</strong>
</div>

<div class="date">
  asked by
  
  <a href="https://stackoverflow.com/users/2555668/usama-zafar" target="_blank" rel="nofollow noopener">
    Usama Zafar
  </a>
  on <a href="https://stackoverflow.com/questions/56054164/using-pytorchs-multiprocessing-along-with-distributed" target="_blank" rel="nofollow noopener">07:22AM - 09 May 19 UTC</a>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

mrshenli: <aside class="quote no-group" data-username="Usama-Zafar" data-post="1" data-topic="65438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/usama-zafar/48/11955_2.png" class="avatar"> Usama-Zafar:</div>
<blockquote>
<p>AssertionError: Default process group is not initialized</p>
</blockquote>
</aside>
<p>above suggests the <code>init_process_group</code> method is not called on the process that tries to use the distributed package. I think the follow line needs to be moved to the <code>run</code> method, and it is the entry point for the spawned process:</p>
<pre><code class="lang-auto"># Initialize Process Group
dist.init_process_group(backend="mpi", group_name="main")
</code></pre>

mrshenli: <aside class="quote no-group quote-modified" data-username="Usama-Zafar" data-post="1" data-topic="65438">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/usama-zafar/48/11955_2.png" class="avatar"> Usama-Zafar:</div>
<blockquote>
<p>dist.init_process_group(backend=“mpi”, group_name=“main”)</p>
</blockquote>
</aside>
<p>BTW, the way you call <code>init_process_group</code> does not look correct to me. You will need to either provide rank+world_size or provide an initialized store. The former will be easier.</p>