PyTorch's `dist_url` init method in Distributed Processing

mbehzad: <p>Hi,</p>
<p>I am running a script for distributed processing on windows with 1 GPU. For that, I am using<br>
<code>torch.distributed</code>. However, I am confused how to set the <code>dist_url</code> flag. Among many others, I went through <a href="https://pytorch.org/docs/stable/distributed.html#environment-variable-initialization" rel="noopener nofollow ugc">this</a> documentation but couldn’t get it.</p>
<p>When I used <code>args.dist_url = 'tcp://localhost:58472'</code> or <code>args.dist_url = 'env://'</code> or even when I manually calculate my IP address and a free port and then set the value as: <code>args.dist_url = "tcp://{}:{}".format(ip,port)</code>, it gives me the following error:</p>
<blockquote>
<p><em><strong>raise RuntimeError(“No rendezvous handler for {}://”.format(result.scheme))</strong></em><br>
<em><strong>RuntimeError: No rendezvous handler for tcp://</strong></em></p>
</blockquote>
<p>Alternatively, I tried to set <code>args.dist_url = "file:///E:/tmp.txt"</code> but then I get the following error:</p>
<blockquote>
<p><em><strong>raise RuntimeError("Distributed package doesn’t have NCCL "</strong></em><br>
<em><strong>RuntimeError: Distributed package doesn’t have NCCL built in</strong></em></p>
</blockquote>
<p>All these errors are raised when the <code>init_process_group()</code> function is called as following:<br>
<code>torch.distributed.init_process_group(backend='nccl', init_method=args.dist_url, world_size=args.world_size, rank=args.rank)</code></p>
<p>Here, note that <code>args.world_size=1</code> and <code>rank=args.rank=0</code>. Any help on this would be appreciated, especially on how to set up the <em><strong>tcp init method</strong></em>.</p>

sio277: <p>Try 127.0.0.1 instead of localhost (e.g. tcp://127.0.0.1:23456).</p>

mbehzad: <p>I tried with that too, but didn’t work.</p>

mbehzad: <p>Following is a simple code which I tried on Ubuntu and it works, but it does not work on Windows. On Windows, it gives the same error (<em><strong>RuntimeError: No rendezvous handler for tcp://</strong></em>) when <code>init_process_group()</code> is called.</p>
<pre><code class="lang-auto">import torch.distributed as dist

dist_url = 'tcp://127.0.0.1:23456'
dist.init_process_group(
   backend = 'nccl',
   rank=0,
   init_method=dist_url,
   world_size=1)
print("Done")
</code></pre>

wayi: <p><a class="mention" href="/u/mrshenli">@mrshenli</a> Do you have any idea why this TCP init method does not work on Windows?</p>

mrshenli: <p>Hey <a class="mention" href="/u/mbehzad">@mbehzad</a>, I noticed you are using NCCL. IIUC, NCCL is not available on Windows. Could you please try gloo instead?</p>
<p>Another question is that which PyTorch version are you using? In v1.7.*, the distributed package only supports FileStore rendezvous on Windows, TCPStore rendezvous is added in v1.8.</p>

mbehzad: <p>Hi <a class="mention" href="/u/mrshenli">@mrshenli</a> you are right indeed. I updated to PyTorch 1.8.1 and used the <code>gloo</code> backend and <code>file</code> <strong>init_method()</strong> and it works.</p>
<p>For anyone wondering, following is a sample code that works on Windows. However, I do not suggest distributed processing on windows.</p>
<pre><code class="lang-auto">import torch.distributed as dist

dist.init_process_group(
   backend = 'gloo',
   rank=0,
   init_method='file:///e:/tmp/some_file',
   world_size=1)

print("Done")
</code></pre>