Timeout in distribuuted init process group

username: Alex_Rak
cooked: <p>I’m try run</p>
<pre><code class="lang-auto">torch.distributed.init_process_group('nccl', world_size=2, rank=0, init_method='file://' + os.path.abspath('./dummy'))
</code></pre>
<p>inside docker container, that’s started with command:</p>
<pre><code class="lang-auto">docker run --runtime=nvidia --network="host" --shm-size 1g -v [from]:[to] -i -t [image] /bin/bash
</code></pre>
<p>But have RuntimeError:</p>
<pre><code class="lang-auto">RuntimeError: Timed out initializing process group in store based barrier on rank: 0, for key: store_based_barrier_key:1
</code></pre>
<p>Can you help me please?</p>

username: H-Huang
cooked: <p>Hello! The timeout is during initialization so it seems that not all workers are joining the group. Two follow up questions:</p>
<ol>
<li>
<p>You are using world_size=2 and you specify rank=0, are you also initializing another worker with rank=1? You can do this by creating multiple processes in a docker container, or running two docker containers and changing the code to use different ranks for each container.</p>
</li>
<li>
<p>If you are using multiple docker containers, since you are using the file init method, you should ensure that both containers can read/write to the same file. You can do this by using a shared volume between containers.</p>
</li>
</ol>

username: Alex_Rak
cooked: <aside class="quote no-group" data-username="H-Huang" data-post="2" data-topic="117166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h-huang/48/35598_2.png" class="avatar"> H-Huang:</div>
<blockquote>
<p>You are using world_size=2 and you specify rank=0, are you also initializing another worker with rank=1?</p>
</blockquote>
</aside>
<p>Yes, I try make it with <a href="https://github.com/facebookresearch/denoiser/blob/master/denoiser/executor.py#L78" rel="noopener nofollow ugc">https://github.com/facebookresearch/denoiser/blob/master/denoiser/executor.py#L78</a> and <a href="https://github.com/facebookresearch/denoiser/blob/master/denoiser/distrib.py#L34" rel="noopener nofollow ugc">https://github.com/facebookresearch/denoiser/blob/master/denoiser/distrib.py#L34</a></p>
<aside class="quote no-group" data-username="H-Huang" data-post="2" data-topic="117166">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h-huang/48/35598_2.png" class="avatar"> H-Huang:</div>
<blockquote>
<p>If you are using multiple docker containers, since you are using the file init method, you should ensure that both containers can read/write to the same file. You can do this by using a shared volume between containers.</p>
</blockquote>
</aside>
<p>No, it’s simple case with only one docker launch.</p>

username: agolynski
cooked: <p>Could you add logging and make sure that both your processes reach the same state before torch.distributed.init_process_group produces an error?</p>
<p>Also, could you make sure that both processes has w access to the directory that contains rendezvous_file and that the file does not exist before you run torch.distributed.init_process_group</p>
<p><a href="https://pytorch.org/docs/stable/distributed.html#shared-file-system-initialization" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/distributed.html#shared-file-system-initialization</a></p>