Update the process group in torch.distributed created using dist.init_process_group

username: adarsh-kr
cooked: <p><strong>Goal</strong>: Distributed Training with <strong>Dynamic</strong> machine location, where worker’s device location can change.</p>
<p><strong>For e.g.</strong> 4 Worker Parameter Server setting. Now, for first 2 epochs 2 workers are run on Machine 1, but after 2 epochs they are supposed to be run on Machine 2.</p>
<p>I am assuming, since the worker’s machine change after 2 epochs, <code>dist.init_process_group()</code> needs to be initialized. However, reinitializing issues this error.</p>
<p><code>RuntimeError: trying to initialize the default process group twice</code></p>
<p>What’s the correct way to update ‘process_group()’ ?</p>
<p>Solution Ideas:</p>
<ol>
<li>Is there anyway to delete the initialized process group? So that before re-initialization using <code>dist.init_process_group()</code> I can delete the prior process group, thus avoiding the issue.</li>
</ol>

username: mrshenli
cooked: <p>Hey <a class="mention" href="/u/adarsh-kr">@adarsh-kr</a>,</p>
<p>There is a <code>destroy_process_group</code> API to clear the default <code>ProcessGroup</code><br>
instance. If you would like to create multiple <code>ProcessGroup</code> instances, you can do so using the <a href="https://pytorch.org/docs/stable/distributed.html#torch.distributed.new_group" rel="nofollow noopener"><code>new_grou</code></a> API.</p>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/05f00532f52883d29a08d96b2961042cc41573ab/torch/distributed/distributed_c10d.py#L530-L577" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/05f00532f52883d29a08d96b2961042cc41573ab/torch/distributed/distributed_c10d.py#L530-L577" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/05f00532f52883d29a08d96b2961042cc41573ab/torch/distributed/distributed_c10d.py#L530-L577</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="530" style="counter-reset: li-counter 529 ;">
<li>def destroy_process_group(group=group.WORLD):</li>
<li>    """</li>
<li>    Destroy a given process group, and deinitialize the distributed package</li>
<li>
</li><li>    Arguments:</li>
<li>        group (ProcessGroup, optional): The process group to be destroyed, if</li>
<li>                                        group.WORLD is given, all process</li>
<li>                                        groups including the default one will</li>
<li>                                        be destroyed.</li>
<li>    """</li>
<li>    global _pg_map</li>
<li>    global _pg_names</li>
<li>    global _pg_group_ranks</li>
<li>    global _default_pg</li>
<li>    global _default_pg_init_method</li>
<li>    global _group_count</li>
<li>
</li><li>    if group == GroupMember.NON_GROUP_MEMBER:</li>
<li>        return</li>
<li>
</li></ol></code></pre>

  This file has been truncated. <a href="https://github.com/pytorch/pytorch/blob/05f00532f52883d29a08d96b2961042cc41573ab/torch/distributed/distributed_c10d.py#L530-L577" target="_blank" rel="nofollow noopener">show original</a>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>