question: NCCL WARN Failed to open libibverbs.so[.1]
context: Just received qty 2 of A6000 and these are not compatible
  with my existing docker file (lack of sm_86 support)
  FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime
  RUN pip install pytorch-lightning==1.0.7
  
  So upgraded my docker to
  FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
  RUN pip install pytorch-lightning==1.5.10
  
  I also made changed to my code the for the lightning braking change from
  trainer = pl.Trainer( gpus=[0,1],  
          distributed_backend='ddp', , 
          ....
  
  to
  trainer = pl.Trainer( gpus=[0,1],  
          strategy='ddp', 
          ....
  
  When I try to train it just stops.  So set env  NCCL_DEBUG=WARN
  as per suggestion
  to get the following output:
  initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
  initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
  ----------------------------------------------------------------------------------------------------
  distributed_backend=nccl
  All distributed processes registered. Starting with 2 processes
  ----------------------------------------------------------------------------------------------------
  60b476048acc:22:22 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
  NCCL version 2.10.3+cuda11.1
  60b476048acc:120:120 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
  
  
  Same happens when I try
  FROM pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime 
  FROM pytorch/pytorch:1.9.1-cuda11.1-cudnn8-runtime 
  FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
  
  My old setup was 2xRTX Titan with nvlink while the new setup is 2xA6000 without a nvlink. nvidia doc says that PCI is used but unclear if I need to do something to use this.
  Distributed communication docs say "NCCL backends are built and included in PyTorch distributed (NCCL only when building with CUDA)" .
  I suspect I am missing something about the breaking changes from pl 1.0 to 1.5. Would appreciate hints as to what to look for.
  Is NCCL something used in pl 1.0 or is this new to pl 1.5?
  Does NCCL need to be installed?
answer: Duplicate of #12235.