Connection refused

antae: <p>hi I implemented tutorial codes in distributed session.<br>
I used node 0 that consisted of two rtx 6000 and node 1 that have a 2080 super.<br>
is it occurred by a mismatch between both nodes?<br>
there are error logs at below.<br>
How can I fixed this problem?</p>
<p>node 0 (master)<br>
python -m torch.distributed.launch --nnode=2 --node_rank=0 --nproc_per_node=1 multi_gpu_pratice/getting_start.py --local_world_size=1</p>
<p>[281611] Initializing process group with: {‘MASTER_ADDR’: ‘163.247.44.175’, ‘MASTER_PORT’: ‘7899’, ‘RANK’: ‘0’, ‘WORLD_SIZE’: ‘2’}<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1&lt;0&gt;<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation</p>
<p>Precision-7920-Tower:281611:281611 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1&lt;0&gt;<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda11.0</p>
<p>node 1<br>
python -m torch.distributed.launch --nnode=2 --node_rank=1 --nproc_per_node=1 multi_gpu_pratice/getting_start.py --local_world_size=1</p>
<p>[8658] world_size = 2, rank = 1, backend=nccl<br>
[8658] rank = 1, world_size = 2, n = 1, device_ids = [0]<br>
MS-7B23:8658:8658 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1&lt;0&gt;<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/IB : No device found.<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1&lt;0&gt;<br>
MS-7B23:8658:8664 [0] NCCL INFO Setting affinity for GPU 0 to 3f<br>
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying<br>
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying</p>

cbalioglu: <p>In your command line I don’t see the <code>--master-addr</code> option which is required for a multi-node training. Do you mind retrying your job with the following?</p>
<pre><code class="lang-auto"># Node 0
pytorch -m torch.distributed.launch --nnode=2 --node_rank=0 --nproc_per_node=1 --master_addr="&lt;hostname_of_rank_0&gt;" multi_gpu_practice/getting_start.py --local_world_size=1

# Node 1
pytorch -m torch.distributed.launch --nnode=2 --node_rank=1 --nproc_per_node=1 --master_addr="&lt;hostname_of_rank_0&gt;" multi_gpu_practice/getting_start.py --local_world_size=1
</code></pre>

antae: <aside class="quote no-group" data-username="antae" data-post="1" data-topic="124008">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/a/ecc23a/48.png" class="avatar"> antae:</div>
<blockquote>
<p>python -m torch.distributed.launch --nnode=2 --node_rank=1 --nproc_per_node=1 multi_gpu_pratice/getting_start.py --local_world_size=1</p>
</blockquote>
</aside>
<p>Thanks for your reply, but <code>--master_addr</code> is setted in code directly.<br>
I initialize <code>NCCL_SOCKET_IFNAME="^eno1,enp0s31f6"</code>, but it just use lo socket.</p>

gcramer23: <blockquote>
<p>Precision-7920-Tower:281611:281611 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1&lt;0&gt;<br>
Precision-7920-Tower:281611:281611 [0] NCCL INFO Using network Socket<br>
NCCL version 2.7.8+cuda11.0</p>
</blockquote>
<blockquote>
<p>MS-7B23:8658:8658 [0] NCCL INFO Bootstrap : Using [0]lo:127.0.0.1&lt;0&gt;<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so).<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/IB : No device found.<br>
MS-7B23:8658:8658 [0] NCCL INFO NET/Socket : Using [0]lo:127.0.0.1&lt;0&gt;<br>
MS-7B23:8658:8664 [0] NCCL INFO Setting affinity for GPU 0 to 3f<br>
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying<br>
MS-7B23:8658:8664 [0] NCCL INFO Call to connect returned Connection refused, retrying</p>
</blockquote>
<p>It looks like NCCL is having a problem establishing a connection. Can you verify that your interfaces are correct. The Common environment variables section provides some information <a href="https://pytorch.org/docs/stable/distributed.html#choosing-the-network-interface-to-use" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>.</p>