Intel E810 RoCE NCCL unhandled system error

BH4CYI: <p>I was trying to start a distributed training across 2 machines using RoCE.<br>
Environment:</p>
<ul>
<li>Ubuntu 20.04</li>
<li>pytorch torch==1.10.1+cu113</li>
</ul>
<p>The task is established by slurm:</p>
<pre><code class="lang-auto">#!/bin/bash
#SBATCH --gres=gpu:2
#SBATCH --nodes=2
#SBATCH --ntasks-per-node=2
#SBATCH --cpus-per-task=4

srun python train.py
</code></pre>
<p>And the error I met is:</p>
<pre><code class="lang-auto">RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.```
</code></pre>
<p>Line 957 in <a href="https://github.com/pytorch/pytorch/blob/v1.10.1/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp" class="inline-onebox" rel="noopener nofollow ugc">pytorch/ProcessGroupNCCL.cpp at v1.10.1 · pytorch/pytorch · GitHub</a> is:</p>
<pre><code class="lang-auto">C10D_NCCL_CHECK(ncclGroupEnd(), c10::nullopt);
</code></pre>
<p>Training works if I set ‘NCCL_IB_DISABLE’ = ‘1’. Which might indicate this might be a RoCE related issue.</p>
<p>I was wonder if I’m looking the correct place. And could anyone help me with this issue?</p>

pritamdamania87: <p>Can you re-run the script with the environment variable <code>NCCL_DEBUG=INFO</code> to see if NCCL debug log reports any errors that could be helpful here?</p>

BH4CYI: <p><code>NCCL_DEBUG=INFO</code> won’t give any useful information. <code>NCCL_DEBUG_SUBSYS=ALL</code> will give some memory alloc information which doesn’t help too much.</p>
<pre><code class="lang-auto">  File "train.py", line 226, in train
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
    ddp_model = DistributedDataParallel(model, device_ids=[local_rank], output_device=local_rank)
  File "/mnt/anaconda/envs/pytorch/lib/python3.7/site-packages/torch/nn/parallel/distributed.py", line 578, in __init__
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7f47040a11d0
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7f47040a1170
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7f47040a11d0
slurm-2:2498277:2498586 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7f47040a1210
    dist._verify_model_across_ranks(self.process_group, parameters)
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:401 Mem Alloc Size 4 pointer 0x7fa25c0a11f0
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:402 Mem Alloc Size 4 pointer 0x7fa25c0a1210
slurm-1:2061438:2061652 [0] NCCL INFO graph/search.cc:452 Mem Alloc Size 4 pointer 0x7fa25c0a1170
RuntimeError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:957, unhandled system error, NCCL version 21.0.3
ncclSystemError: System call (socket, malloc, munmap, etc) failed.
</code></pre>

BH4CYI: <p>I did get a warning saying <code>libibverbs: Warning: couldn't load driver 'libi40iw-rdmav25.so': libi40iw-rdmav25.so: cannot open shared object file: No such file or directory</code></p>
<p>But as far as I know libi40iw is for Intel Ethernet Connection X722 RDMA. And my NIC is E810, which doesn’t compatible with this lib.</p>

BH4CYI: <p>I rerun the code today and found such INFO:</p>
<pre><code class="lang-auto">misc/ibvwrap.cc:284 NCCL WARN Call to ibv_modify_qp failed with error Invalid argument
</code></pre>

pritamdamania87: <p><a class="mention" href="/u/bh4cyi">@BH4CYI</a> Might be useful to directly contact the NCCL team here to debug this further. I’d suggest probably creating a gh issue here: <a href="https://github.com/NVIDIA/nccl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/nccl: Optimized primitives for collective multi-GPU communication</a> and sharing all the NCCL debug logs with them to determine a resolution.</p>

BH4CYI: <p>GitHub issue is here: <a href="https://github.com/NVIDIA/nccl/issues/622" class="inline-onebox" rel="noopener nofollow ugc">Intel E810 ncclSystemError: System call (socket, malloc, munmap, etc) failed. · Issue #622 · NVIDIA/nccl · GitHub</a></p>

BH4CYI: <p>I applyed the patch mentioned in the github issue, and nccl-test passed on multi node. I think what I can do now is to re-compile PyTorch with patched NCCL.</p>