Can not realize distributed training across machines with DDP

username: KaiiZhang
cooked: <h1><a name="backgroud-1" class="anchor" href="#backgroud-1"></a>Backgroud</h1>
<p>Hi, i’m new to distributed training across machines with DDP. I’m currently writing a simple demo to try it out. My demo can run normally on a single-machine-multiple-GPUs paradigm, but it cannot be applied to multiple-machines. I tried many ways but still can’t solve this problem. I really need some help, thanks.</p>
<h1><a name="environment-2" class="anchor" href="#environment-2"></a>Environment</h1>
<ul>
<li>2 machine, each with 2 RTX3090</li>
<li>python==3.6.13</li>
<li>pytorch==1.8.0+cu111</li>
<li>CUDA==11.1</li>
</ul>
<p>I have tested in other environment but still facing the same problem:</p>
<ul>
<li>pytorch==1.7.0+cu101</li>
<li>pytorch==1.8.1+cu111</li>
<li>pytorch==1.9.0+cu111</li>
</ul>
<h1><a name="code-3" class="anchor" href="#code-3"></a>Code</h1>
<p>I uploaded my code to GitHub. There are two versions with different launch methods:</p>
<ul>
<li><a href="https://github.com/KaiiZhang/DDP_demo/blob/main/mnist-ddp-tcp.py" rel="noopener nofollow ugc">mnist-ddp-tcp.py</a></li>
<li><a href="https://github.com/KaiiZhang/DDP_demo/blob/main/mnist-ddp-env.py" rel="noopener nofollow ugc">mnist-ddp-env.py</a></li>
</ul>
<p>I use the TCP version (mnist-ddp-tcp.py) as an example:</p>
<h2><a name="successfully-run-on-single-machine-multi-gpus-4" class="anchor" href="#successfully-run-on-single-machine-multi-gpus-4"></a>Successfully run on single-machine-multi-GPUs</h2>
<pre><code class="lang-auto"># bash
python mnist-ddp-tcp.py --init_method tcp://[MACHINE0_IP]:[PORT] --rank 0 --world_size 2 --gpuid 0 
python mnist-ddp-tcp.py --init_method tcp://[MACHINE0_IP]:[PORT] --rank 1 --world_size 2 --gpuid 1
</code></pre>
<h2><a name="failed-to-run-on-multi-machines-5" class="anchor" href="#failed-to-run-on-multi-machines-5"></a>Failed to run on multi-machines</h2>
<pre><code class="lang-auto"># bash machine-0
python mnist-ddp-tcp.py --init_method tcp://[MACHINE0_IP]:[PORT] --rank 0 --world_size 2 --gpuid 0
# bash machine-1
python mnist-ddp-tcp.py --init_method tcp://[MACHINE0_IP]:[PORT] --rank 1 --world_size 2 --gpuid 0

</code></pre>
<p><strong>The program is blocked in L78: dist.barrier(device_ids=[gpu]).</strong> When I comment out L78, the program gets stuck at L98.</p>
<h3><a name="machine0-6" class="anchor" href="#machine0-6"></a>machine0</h3>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/a/1a7868c2aebb2cdd3e000afde7573362b98809ca.png" data-download-href="https://discuss.pytorch.org/uploads/default/1a7868c2aebb2cdd3e000afde7573362b98809ca" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a7868c2aebb2cdd3e000afde7573362b98809ca_2_690x58.png" alt="image" data-base62-sha1="3MaoanyFsC6T5l3isFxzVUpEs1c" width="690" height="58" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a7868c2aebb2cdd3e000afde7573362b98809ca_2_690x58.png, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a7868c2aebb2cdd3e000afde7573362b98809ca_2_1035x87.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/1/a/1a7868c2aebb2cdd3e000afde7573362b98809ca_2_1380x116.png 2x" data-dominant-color="272727"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1656×140 11.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<h2><a name="possible-causes-7" class="anchor" href="#possible-causes-7"></a>Possible causes</h2>
<p>I speculate that the problem might be with the firewalls of my two machines. But I also observed that the L77 is printed normally, that is, my process group is initialized normally (L74).  So I don’t know how to find and fix the problem.</p>

username: cbalioglu
cooked: <p>Looks like you are using NCCL as your process group backend? Do you mind giving it a try with Gloo?</p>

username: KaiiZhang
cooked: <p>Yes, i have been using NCCL before.<br>
Now I tried gloo and the program works fine. Thank you for your advice.</p>
<h2>
<a name="note-1" class="anchor" href="#note-1"></a>Note</h2>
<p>I tried gloo and at the beginning I got “address family mismatch” error that is same as <a href="https://discuss.pytorch.org/t/runtimeerror-address-family-mismatch-when-use-gloo-backend/64753">discuss.64753</a>. I solved this by specifying GLOO_SOCKET_IFNAME.</p>
<pre><code class="lang-auto"># bash
export GLOO_SOCKET_IFNAME=eno2np1
</code></pre>
<p>Inspired by this, i also tried specifying NCCL_SOCKET_IFNAME, but that didn’t fix the nccl problem.</p>
<pre><code class="lang-auto"># bash
export NCCL_SOCKET_IFNAME=eno2np1
</code></pre>
<p><strong>So I still can’t use nccl at the moment. Gloo’s lack of support for some operators limits the functionality in the code. Could you please give me some advice on how to solve this problem?</strong></p>

username: KaiiZhang
cooked: <h2><a name="supplement-1" class="anchor" href="#supplement-1"></a>Supplement</h2>
<p>When I use nccl backend, machine0 is blocked in L78: dist.barrier(device_ids=[gpu]). After a period of time, machine1 will automatically terminate with the following error:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a95d7165d5f958085961b00d7e3af4a711ea3c9.png" data-download-href="https://discuss.pytorch.org/uploads/default/6a95d7165d5f958085961b00d7e3af4a711ea3c9" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/6/a/6a95d7165d5f958085961b00d7e3af4a711ea3c9.png" alt="image" data-base62-sha1="fcTDRCWMumaMOeQRMkxk7aNkbzP" width="690" height="164" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1008×240 14.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Yanli_Zhao
cooked: <p><a class="mention" href="/u/kaiizhang">@KaiiZhang</a> looks like both ranks passed the init_process_group() and they failed at dist.barrier() step in your codes?</p>

username: KaiiZhang
cooked: <p>Yes, that is what I think is confusing.<br>
In fact, when the environment is changed to “ pytorch==1.9.0+cu111”, process-1 is able to pass the dist.barrier() , and then gets stuck at the next synchronization point.  Process-0 is still stuck in the barrier.</p>

username: taehyunzzz
cooked: <p>Did you come up with any solution to this?<br>
I’m wondering if this could be some TCP issue, ie the two remote server nodes just cannot communicate for some reason… <img src="https://discuss.pytorch.org/images/emoji/apple/frowning_face.png?v=12" title=":frowning_face:" class="emoji" alt=":frowning_face:" loading="lazy" width="20" height="20"></p>

username: taehyunzzz
cooked: <p>I think the problem is your firewall. Try <a href="https://discuss.pytorch.org/t/connection-refused-with-gloo-process-group-initialization/144857/6">this</a>.</p>

username: pritamdamania87
cooked: <p><a class="mention" href="/u/kaiizhang">@KaiiZhang</a> Can you run with the environment variable <code>NCCL_DEBUG=INFO</code>, that would give more information about why NCCL is failing. Could you share the entire logs for all processes after setting <code>NCCL_DEBUG=INFO</code>.</p>