A6000 NCCL WARN Failed to open libibverbs.so

username: John_Grabner
cooked: <p><a href="https://pytorch.org/docs/stable/distributed.html" rel="noopener nofollow ugc">pytorch-lightning</a> say “NCCL backends are built and included in PyTorch distributed (NCCL only when building with CUDA)”</p>
<p>When I upgrade my driver and docker from</p>
<blockquote>
<p>nvidia-driver-470 on the host<br>
FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime</p>
</blockquote>
<p>to</p>
<blockquote>
<p>nvidia-driver-510 on the host<br>
FROM pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime<br>
or later</p>
</blockquote>
<p>I get the error<br>
misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]</p>
<p>I just upgraded my 2x (Titan RTX with nvlink) to 2x (A600 with no nvlink), pl version pytorch-lightning==1.0.7 and same error for version 1.5 .</p>
<p>My call to train is as follows:</p>
<pre><code class="lang-auto">trainer = pl.Trainer( gpus=[0,1],  
        distributed_backend='ddp', # strategy='ddp' for pl=1.5
        ...
</code></pre>
<p>I can’t stay on pytorch:1.6.0-cuda10.1-cudnn7 since A6000 needs sm_86 support.</p>
<p>Is there something else I need to do when going from the docker image of pytorch:1.6.0-cuda10.1-cudnn7 to pytorch:1.7.1-cuda11.0-cudnn8-runtime or later to use ddp?</p>
<h3><a name="demo-of-issue-1" class="anchor" href="#demo-of-issue-1"></a>Demo of Issue</h3>
<aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/344;"><img src="https://opengraph.githubassets.com/b061b9f06c969dda4fa220470610d3e79e02d9caebd0c0039c0d80729131dff1/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" target="_blank" rel="noopener nofollow ugc">GitHub - johngrabner/NCCL-WARN-Failed-to-open-libibverbs</a></h3>

  <p>Contribute to johngrabner/NCCL-WARN-Failed-to-open-libibverbs development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<h3><a name="versions-2" class="anchor" href="#versions-2"></a>Versions</h3>
<p>docker image pytorch/pytorch:1.7.1-cuda11.0-cudnn8-runtime</p>
<p>bug_1 | Collecting environment information…<br>
bug_1 | PyTorch version: 1.10.0<br>
bug_1 | Is debug build: False<br>
bug_1 | CUDA used to build PyTorch: 11.3<br>
bug_1 | ROCM used to build PyTorch: N/A<br>
bug_1 | OS: Ubuntu 18.04.6 LTS (x86_64)<br>
bug_1 | GCC version: Could not collect<br>
bug_1 | Clang version: Could not collect<br>
bug_1 | CMake version: Could not collect<br>
bug_1 | Libc version: glibc-2.17<br>
bug_1 | Python version: 3.7.11 (default, Jul 27 2021, 14:32:16) [GCC 7.5.0] (64-bit runtime)<br>
bug_1 | Python platform: Linux-5.4.0-100-generic-x86_64-with-debian-buster-sid<br>
bug_1 | Is CUDA available: True<br>
bug_1 | CUDA runtime version: Could not collect<br>
bug_1 | GPU models and configuration:<br>
bug_1 | GPU 0: NVIDIA RTX A6000<br>
bug_1 | GPU 1: NVIDIA RTX A6000<br>
bug_1 |<br>
bug_1 | Nvidia driver version: 510.47.03<br>
bug_1 | cuDNN version: Could not collect<br>
bug_1 | HIP runtime version: N/A<br>
bug_1 | MIOpen runtime version: N/A<br>
bug_1 | Is XNNPACK available: True<br>
bug_1 | Versions of relevant libraries:<br>
bug_1 | [pip3] numpy==1.21.2<br>
bug_1 | [pip3] pytorch-lightning==1.5.10<br>
bug_1 | [pip3] torch==1.10.0<br>
bug_1 | [pip3] torchelastic==0.2.0<br>
bug_1 | [pip3] torchmetrics==0.7.2<br>
bug_1 | [pip3] torchtext==0.11.0<br>
bug_1 | [pip3] torchvision==0.11.0<br>
bug_1 | [conda] blas 1.0 mkl<br>
bug_1 | [conda] cudatoolkit 11.3.1 ha36c431_9 nvidia<br>
bug_1 | [conda] ffmpeg 4.3 hf484d3e_0 pytorch<br>
bug_1 | [conda] mkl 2021.3.0 h06a4308_520<br>
bug_1 | [conda] mkl-service 2.4.0 py37h7f8727e_0<br>
bug_1 | [conda] mkl_fft 1.3.1 py37hd3c417c_0<br>
bug_1 | [conda] mkl_random 1.2.2 py37h51133e4_0<br>
bug_1 | [conda] numpy 1.21.2 py37h20f2e39_0<br>
bug_1 | [conda] numpy-base 1.21.2 py37h79a1101_0<br>
bug_1 | [conda] pytorch 1.10.0 py3.7_cuda11.3_cudnn8.2.0_0 pytorch<br>
bug_1 | [conda] pytorch-lightning 1.5.10 pypi_0 pypi<br>
bug_1 | [conda] pytorch-mutex 1.0 cuda pytorch<br>
bug_1 | [conda] torchelastic 0.2.0 pypi_0 pypi<br>
bug_1 | [conda] torchmetrics 0.7.2 pypi_0 pypi<br>
bug_1 | [conda] torchtext 0.11.0 py37 pytorch<br>
bug_1 | [conda] torchvision 0.11.0 py37_cu113 pytorch<br>
bug_1 | ****************************<br>
bug_1 | initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2<br>
bug_1 | /opt/conda/lib/python3.7/site-packages/pytorch_lightning/trainer/configuration_validator.py:120: UserWarning: You passed in a <code>val_dataloader</code> but have no <code>validation_step</code>. Skipping val loop.<br>
bug_1 | rank_zero_warn(“You passed in a <code>val_dataloader</code> but have no <code>validation_step</code>. Skipping val loop.”)<br>
bug_1 | initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2<br>
bug_1 | ----------------------------------------------------------------------------------------------------<br>
bug_1 | distributed_backend=nccl<br>
bug_1 | All distributed processes registered. Starting with 2 processes<br>
bug_1 | ----------------------------------------------------------------------------------------------------<br>
bug_1 |<br>
bug_1 |<br>
bug_1 | be7823cc7810:7:7 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]<br>
bug_1 | NCCL version 2.10.3+cuda11.3<br>
bug_1 |<br>
bug_1 | be7823cc7810:72:72 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]</p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="John_Grabner" data-post="1" data-topic="145630">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/john_grabner/48/26165_2.png" class="avatar"> John_Grabner:</div>
<blockquote>
<p>misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]</p>
</blockquote>
</aside>
<p>This is not necessarily an error and just a warning that NCCL couldn’t find libibverbs.so and as result things like Infiniband connectivity might be affected. Are you using something like Infiniband or GPU RDMA in your setup?</p>

username: John_Grabner
cooked: <p>Just 2 A6000 in PCIe slots. Waiting for nvlink.<br>
I create a small demo of this issue and it just sits there.</p><aside class="onebox allowlistedgeneric" data-onebox-src="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs">
  <header class="source">
      <img src="https://github.githubassets.com/favicons/favicon.svg" class="site-icon" width="32" height="32">

      <a href="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" target="_blank" rel="noopener nofollow ugc">GitHub</a>
  </header>

  <article class="onebox-body">
    <div class="aspect-image" style="--aspect-ratio:690/345;"><img src="https://opengraph.githubassets.com/b061b9f06c969dda4fa220470610d3e79e02d9caebd0c0039c0d80729131dff1/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" class="thumbnail" width="690" height="345"></div>

<h3><a href="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs" target="_blank" rel="noopener nofollow ugc">GitHub - johngrabner/NCCL-WARN-Failed-to-open-libibverbs</a></h3>

  <p>Contribute to johngrabner/NCCL-WARN-Failed-to-open-libibverbs development by creating an account on GitHub.</p>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>My real code just sits there too when I use 2 GPU and ddp.  With Titan RTX this code was stable for a year.  If I just one one GPU it works.</p>

username: pritamdamania87
cooked: <p>This might be a pytorch lightning setup issue, can you create an issue in <a href="https://github.com/PyTorchLightning/pytorch-lightning" class="inline-onebox" rel="noopener nofollow ugc">GitHub - PyTorchLightning/pytorch-lightning: The lightweight PyTorch wrapper for high-performance AI research. Scale your models, not the boilerplate.</a> to confirm whether it is indeed related to PyTorch Lightning?</p>
<p>Btw does your script exit or just get stuck after printing those warnings? If it is the latter, I think it might be some sort of setup issue related to PyTorch Lightning.</p>

username: John_Grabner
cooked: <p>The demo code runs to completion with 1 GPU and no dpp in about 2 minutes on A6000.</p>
<p>My real code is too big, so I created a very short demonstration of this issue at <a href="https://github.com/johngrabner/NCCL-WARN-Failed-to-open-libibverbs.git" rel="noopener nofollow ugc">small demo code</a>.</p>
<p>Since my big code ran correctly with dpp using TITAN RTX on pytorch 1.6, and my code fails in exactly the same way as the demo with pytorch 1.7, it looks like and issue with either NVIDIA or pytorch. But I did raise an  <a href="https://github.com/PyTorchLightning/pytorch-lightning/issues/12235" rel="noopener nofollow ugc">item with pytorch-lightining</a></p>

username: John_Grabner
cooked: <p>Does NCCL work on all PCIE setups (ie: all motherboards)?<br>
My motherboard is Gigabyte DesignAre.</p>

username: ptrblck
cooked: <p>The issue is also cross-posted <a href="https://github.com/pytorch/pytorch/issues/73790#issuecomment-1059662412">here</a> and so far I’m unable to reproduce the hang.<br>
Let’s wait for the <code>nccl-test</code> runs on your setup to see if NCCL is working at all or what is causing the hang.</p>

username: John_Grabner
cooked: <p><a href="https://discuss.pytorch.org/u/ptrblck">ptrblck</a> Thank you for suggesting to disable IOMMU.  IOMMU=on did cause my system to lock up. You are so kind to offer your help to everyone. Thank you.</p>

username: ptrblck
cooked: <p>Good to hear it’s working now! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>