NCCL WARN Failed to open libibverbs.so[.1]
Just received qty 2 of A6000 and these are not compatible
with my existing docker file (lack of sm_86 support)
FROM pytorch/pytorch:1.6.0-cuda10.1-cudnn7-runtime
RUN pip install pytorch-lightning==1.0.7

So upgraded my docker to
FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime
RUN pip install pytorch-lightning==1.5.10

I also made changed to my code the for the lightning braking change from
trainer = pl.Trainer( gpus=[0,1],  
        distributed_backend='ddp', , 
        ....

to
trainer = pl.Trainer( gpus=[0,1],  
        strategy='ddp', 
        ....

When I try to train it just stops.  So set env  NCCL_DEBUG=WARN
as per suggestion
to get the following output:
initializing distributed: GLOBAL_RANK: 0, MEMBER: 1/2
initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2
----------------------------------------------------------------------------------------------------
distributed_backend=nccl
All distributed processes registered. Starting with 2 processes
----------------------------------------------------------------------------------------------------
60b476048acc:22:22 [0] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]
NCCL version 2.10.3+cuda11.1
60b476048acc:120:120 [1] misc/ibvwrap.cc:63 NCCL WARN Failed to open libibverbs.so[.1]


Same happens when I try
FROM pytorch/pytorch:1.8.1-cuda11.1-cudnn8-runtime 
FROM pytorch/pytorch:1.9.1-cuda11.1-cudnn8-runtime 
FROM pytorch/pytorch:1.10.0-cuda11.3-cudnn8-runtime

My old setup was 2xRTX Titan with nvlink while the new setup is 2xA6000 without a nvlink. nvidia doc says that PCI is used but unclear if I need to do something to use this.
Distributed communication docs say "NCCL backends are built and included in PyTorch distributed (NCCL only when building with CUDA)" .
I suspect I am missing something about the breaking changes from pl 1.0 to 1.5. Would appreciate hints as to what to look for.
Is NCCL something used in pl 1.0 or is this new to pl 1.5?
Does NCCL need to be installed?
Duplicate of #12235.