Single node 2 GPU distributed training nccl-backend hanged

Chenchao_Zhao: <p>I tried to train MNIST using <code>torch.distributed.launch</code> nccl backend</p>
<p>The launch command</p>
<pre><code class="lang-auto">export NCCL_DEBUG=INFO
export NCCL_IB_DISABLE=true  # use or not does not change the results
echo "NCCL_IB_DISABLE=$NCCL_IB_DISABLE"
export NCCL_SOCKET_IFNAME=eno1,eth0  # use or not does not change the results

python3 -m torch.distributed.launch --nproc_per_node 2 \
                                   --nnodes 1 \
                                   --node_rank 0 \
                                   --master_addr="0.0.0.0" \
                                   --master_port=2333 \
                                   main.py \
                                   --epochs 3 \
                                   --lr 1e-3 \
                                   --batch_size 150
</code></pre>
<ol>
<li>
<code>gloo</code> backend works just fine</li>
<li>
<code>nccl</code> got stuck</li>
<li>i have tried suggestions on the forum but none of them worked</li>
</ol>
<p>debug info:</p>
<pre><code class="lang-auto">sh start-dist-train.sh
NCCL_IB_DISABLE=true
*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
nccl
nccl
torch-research-2gpu-0:48249:48249 [0] NCCL INFO Bootstrap : Using [0]eth0:10.244.26.37&lt;0&gt;
torch-research-2gpu-0:48249:48249 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

torch-research-2gpu-0:48249:48249 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
torch-research-2gpu-0:48249:48249 [0] NCCL INFO NET/Socket : Using [0]eth0:10.244.26.37&lt;0&gt;
torch-research-2gpu-0:48249:48249 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda10.2
torch-research-2gpu-0:48250:48250 [1] NCCL INFO Bootstrap : Using [0]eth0:10.244.26.37&lt;0&gt;
torch-research-2gpu-0:48250:48250 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation

torch-research-2gpu-0:48250:48250 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
torch-research-2gpu-0:48250:48250 [1] NCCL INFO NET/Socket : Using [0]eth0:10.244.26.37&lt;0&gt;
torch-research-2gpu-0:48250:48250 [1] NCCL INFO Using network Socket
</code></pre>

cbalioglu: <p>Hey Chenchao,</p>
<p>Couple questions:</p>
<ul>
<li>Which PyTorch version are you using?</li>
<li>What is your OS/GPU setup?</li>
<li>Is it possible to share your script?</li>
<li>Is this behavior reproducible at every run?</li>
</ul>

Chenchao_Zhao: <p>Hey Can,</p>
<ul>
<li>pytorch version 1.8.1-cu102</li>
<li>the instance is kubeflow notebook server</li>
<li>container image is ubuntu:20.04</li>
<li>behavior is reproducible</li>
</ul>
<p>I fixed the issue by setting master ip to localhost<br>
<code>export NCCL_SOCKET_IFNAME =lo</code></p>
<p>I haven’t tried a multi-node training with several GPUs per node yet.</p>

cbalioglu: <p>Cool, glad that you could fix the problem. Let us know if you experience any issues with a multi-node setup.</p>