Hanging distributed data parallel on interactive example

kirk86: <p>Anybody can clarify why I keep getting cuda error when using torch.nn.parallel.DistributedDataParallel?</p>
<p>MWE:</p>
<pre><code class="lang-auto">if torch.cuda.device_count() &gt; 1:
    torch.distributed.init_process_group(backend='nccl')
    model = torch.nn.parallel.DistributedDataParallel(model).to(device)
</code></pre>
<p>executing script:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch main.py
</code></pre>
<p>Am I using it incorrectly or am I missing something else?</p>
<p>A MWE:</p>
<pre><code class="lang-auto">import os
import torch
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '12355'
torch.distributed.init_process_group(backend='nccl', rank=1, world_size=2)
</code></pre>
<p>Then the universe hangs completely:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/0/021a116dc95983c8899b9e67aa557c88ed6ee94c.png" data-download-href="https://discuss.pytorch.org/uploads/default/021a116dc95983c8899b9e67aa557c88ed6ee94c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/0/021a116dc95983c8899b9e67aa557c88ed6ee94c_2_690x59.png" alt="image" data-base62-sha1="iANWwBCbqaHHBXam45qkfhBU9S" width="690" height="59" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/0/021a116dc95983c8899b9e67aa557c88ed6ee94c_2_690x59.png, https://discuss.pytorch.org/uploads/default/original/2X/0/021a116dc95983c8899b9e67aa557c88ed6ee94c.png 1.5x, https://discuss.pytorch.org/uploads/default/original/2X/0/021a116dc95983c8899b9e67aa557c88ed6ee94c.png 2x" data-dominant-color="1C201D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">867×75 28.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Terminal is not released back to the user. Any ideas why?</p>

pietern: <p>You have to use <code>DistributedDataParallel</code> with as many processes as the value of <code>world_size</code>. If you specify <code>rank=1</code> on a single process, it will hang around waiting for the process with <code>rank=0</code> to start.</p>
<p>Check out the <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" rel="nofollow noopener">launch utility</a> for easy launching of multiple processes. You can use it like this:</p>
<pre><code class="lang-auto">python -m torch.distributed.launch --nproc_per_node=2 ./my_script.py
</code></pre>

kirk86: <p>Awesome, thanks so much for clarifying that!<br>
If you don’t mind me asking for some additional clarifications?</p>
<aside class="quote no-group" data-username="pietern" data-post="2" data-topic="55439">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pietern/48/4981_2.png" class="avatar"> pietern:</div>
<blockquote>
<p>Check out the <a href="https://pytorch.org/docs/stable/distributed.html#launch-utility" rel="noopener nofollow ugc">launch utility</a> for easy launching of multiple processes. You can use it like this</p>
</blockquote>
</aside>
<p>The lauch utility has to be used in combination with <code>DistributedDataParallel</code> right, or can it be used on its own?</p>
<p>Using <code>DistributedDataParallel</code> does it mean that it has to be used in combination with <code>torch.utils.data.distributed.DistributedSampler</code>?</p>
<p>Thanks again!</p>

kirk86: <p>The correct call for multi-gpu single node is:<br>
<code>world_size = 1</code>  &lt;-- number of nodes<br>
<code>rank=0</code>  &lt;-- which node is running which process</p>