RuntimeError: NCCL Error 3: internal error

William_Ravenscroft: <p>Hi,<br>
I am using DataParallel in a SGE enviroment with cudatoolkit 10.1 and NCCL version 2.7. I get the following error</p>
<pre><code class="lang-auto">/miniconda3/envs/cs21/lib/python3.8/site-packages/torch/nn/parallel/comm.py", line 56, in broadcast_coalesced
    return torch._C._broadcast_coalesced(tensors, devices, buffer_size)

RuntimeError: NCCL Error 3: internal error
</code></pre>
<p>The NCCL debug info is as follows</p>
<pre><code class="lang-auto">node20:22347:22347 [0] NCCL INFO Bootstrap : Using [0]ib0:10.10.9.220&lt;0&gt;
node20:22347:22347 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
node20:22347:22347 [0] NCCL INFO NET/IB : Using [0]qib0:1/IB ; OOB ib0:10.10.9.220&lt;0&gt;
node20:22347:22347 [0] NCCL INFO Using network IB
NCCL version 2.7.8+cuda10.1

node20:22347:22436 [1] graph/xml.h:77 NCCL WARN Attribute class of node nic not found
node20:22347:22436 [1] NCCL INFO graph/topo.cc:312 -&gt; 3
node20:22347:22436 [1] NCCL INFO graph/topo.cc:348 -&gt; 3
node20:22347:22436 [1] NCCL INFO graph/topo.cc:395 -&gt; 3
node20:22347:22436 [1] NCCL INFO graph/topo.cc:467 -&gt; 3
node20:22347:22436 [1] NCCL INFO graph/topo.cc:570 -&gt; 3

node20:22347:22435 [0] graph/xml.h:77 NCCL WARN Attribute class of node nic not found
node20:22347:22436 [1] NCCL INFO init.cc:581 -&gt; 3
node20:22347:22435 [0] NCCL INFO graph/topo.cc:312 -&gt; 3
node20:22347:22436 [1] NCCL INFO init.cc:840 -&gt; 3
node20:22347:22435 [0] NCCL INFO graph/topo.cc:348 -&gt; 3
node20:22347:22435 [0] NCCL INFO graph/topo.cc:395 -&gt; 3
node20:22347:22435 [0] NCCL INFO graph/topo.cc:467 -&gt; 3
node20:22347:22435 [0] NCCL INFO graph/topo.cc:570 -&gt; 3
node20:22347:22435 [0] NCCL INFO init.cc:581 -&gt; 3
node20:22347:22435 [0] NCCL INFO init.cc:840 -&gt; 3
node20:22347:22436 [1] NCCL INFO group.cc:73 -&gt; 3 [Async thread]
node20:22347:22435 [0] NCCL INFO group.cc:73 -&gt; 3 [Async thread]
node20:22347:22347 [0] NCCL INFO init.cc:906 -&gt; 3
</code></pre>
<p>Does anyone have any idea what this issue might be caused by?</p>
<p>Cheers in advance</p>

pritamdamania87: <aside class="quote no-group" data-username="William_Ravenscroft" data-post="1" data-topic="113139">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/william_ravenscroft/48/35149_2.png" class="avatar"> William_Ravenscroft:</div>
<blockquote>
<pre><code class="lang-auto">RuntimeError: NCCL Error 3: internal error
</code></pre>
</blockquote>
</aside>
<p>NCCL error 3 seems to be either a bug in NCCL or some memory corruption: <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/api/types.html#c.ncclInternalError" class="inline-onebox" rel="noopener nofollow ugc">Types — NCCL 2.19.3 documentation</a>. Maybe you can create an issue at <a href="https://github.com/NVIDIA/nccl" class="inline-onebox" rel="noopener nofollow ugc">GitHub - NVIDIA/nccl: Optimized primitives for collective multi-GPU communication</a> to see if the NCCL team has some guidelines on how to debug this.</p>

William_Ravenscroft: <p>To follow up, I think I actually had 2 issues firstly I had to set</p>
<pre><code class="lang-auto">export NCCL_SOCKET_IFNAME=&lt;VALUE&gt;
export NCCL_IB_DISABLE=1
</code></pre>
<p>Replacing  with your relevant interface  - use the <code>ifconfig</code> to find it. And I think my second issue was using a dataloader with multiple workers but I hadn’t allocated enough processes to the job in my job submission.</p>

SM2023: <p>Hi <a class="mention" href="/u/william_ravenscroft">@William_Ravenscroft</a> , I always run my code successfully, but today when I run it gave me this error in the same condition.<br>
would you please help me with that/<br>
what is</p>
<p>export NCCL_SOCKET_IFNAME=<br>
?</p>

William_Ravenscroft: <p>Refer to the Nvidia documentation: <a href="https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html#nccl-socket-ifname" class="inline-onebox" rel="noopener nofollow ugc">Environment Variables — NCCL 2.18.1 documentation</a></p>