Pytorch version incompatible with cuda

r00bi: <p>I’ve got this error after running the code written with PyTorch on two GPUs:<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.</p>
<p>torch version: ‘1.11.0+cu102’<br>
cuda version:  11.4    driver: 470.63.01</p>
<p>Could you please tell if this error happened because of incompatibility of cuda and torch version? and how can I fix it?<br>
Thanks!</p>

ptrblck: <p>Could you rerun your script with <code>NCCL_DEBUG=INFO</code> and post the log here, please?</p>

r00bi: <p>Here is the log:<br>
trainer.fit(model, train_loader.train_dataloader(), val_loader.val_dataloader())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, **kwargs)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1138, in _run<br>
self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1435, in _call_setup_hook<br>
self.training_type_plugin.barrier(“pre_setup”)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py”, line 403, in barrier<br>
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 2776, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
~</p>

ptrblck: <p>This output doesn’t show any NCCL logs but just the PyTorch stacktrace, so make sure to <code>export</code> this env variable before rerunning your script.</p>

r00bi: <p>I think this is NCCL logs:<br>
c309-002:278034:278034 [0] enqueue.cc:102 NCCL WARN Cuda failure ‘invalid device function’<br>
c309-002:278034:278034 [0] NCCL INFO Bootstrap : Using ib0:192.168.41.194&lt;0&gt;<br>
c309-002:278034:278034 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
c309-002:278034:278034 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:192.168.41.194&lt;0&gt;<br>
c309-002:278034:278034 [0] NCCL INFO Using network IB<br>
NCCL version 2.10.3+cuda10.2</p>
<p>c309-002:278123:278123 [1] enqueue.cc:102 NCCL WARN Cuda failure ‘invalid device function’<br>
c309-002:278123:278123 [1] NCCL INFO Bootstrap : Using ib0:192.168.41.194&lt;0&gt;<br>
c309-002:278123:278123 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
c309-002:278123:278123 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:192.168.41.194&lt;0&gt;<br>
c309-002:278123:278123 [1] NCCL INFO Using network IB<br>
c309-002:278034:278243 [0] NCCL INFO Channel 00/02 :    0   1<br>
c309-002:278123:278245 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;0<br>
c309-002:278034:278243 [0] NCCL INFO Channel 01/02 :    0   1<br>
c309-002:278123:278245 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,00000000,00000000<br>
c309-002:278034:278243 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
c309-002:278034:278243 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff<br>
c309-002:278123:278245 [1] NCCL INFO Channel 00 : 1[81000] → 0[21000] via P2P/IPC<br>
c309-002:278123:278245 [1] NCCL INFO Channel 01 : 1[81000] → 0[21000] via P2P/IPC<br>
c309-002:278034:278243 [0] NCCL INFO Channel 00 : 0[21000] → 1[81000] via P2P/IPC<br>
c309-002:278034:278243 [0] NCCL INFO Channel 01 : 0[21000] → 1[81000] via P2P/IPC<br>
c309-002:278123:278245 [1] NCCL INFO Connected all rings<br>
c309-002:278123:278245 [1] NCCL INFO Connected all trees<br>
c309-002:278034:278243 [0] NCCL INFO Connected all rings<br>
c309-002:278034:278243 [0] NCCL INFO Connected all trees<br>
c309-002:278123:278245 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
c309-002:278123:278245 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
c309-002:278034:278243 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
c309-002:278034:278243 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
c309-002:278123:278245 [1] NCCL INFO comm 0x154ba4001240 rank 1 nranks 2 cudaDev 1 busId 81000 - Init COMPLETE<br>
c309-002:278034:278243 [0] NCCL INFO comm 0x14c2d0001240 rank 0 nranks 2 cudaDev 0 busId 21000 - Init COMPLETE<br>
c309-002:278034:278034 [0] NCCL INFO Launch mode Parallel</p>
<p>c309-002:278123:278123 [1] enqueue.cc:300 NCCL WARN Cuda failure ‘invalid device function’<br>
c309-002:278123:278123 [1] NCCL INFO group.cc:347 → 1</p>
<p>c309-002:278034:278034 [0] enqueue.cc:300 NCCL WARN Cuda failure ‘invalid device function’<br>
c309-002:278034:278034 [0] NCCL INFO group.cc:347 → 1<br>
~</p>

ptrblck: <p>Yes, these are the NCL logs, but they also don’t show any previous error besides:</p>
<pre><code class="lang-python">enqueue.cc:102 NCCL WARN Cuda failure ‘invalid device function’
</code></pre>
<p>Try to rerun the code via additional debugging variables:</p>
<pre><code class="lang-python">TORCH_DISTRIBUTED_DEBUG=INFO
TORCH_SHOW_CPP_STACKTRACES=1
</code></pre>

r00bi: <p>trainer.fit(model, train_loader.train_dataloader(), val_loader.val_dataloader())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, <em><em>kwargs)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1138, in _run<br>
self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1435, in _call_setup_hook<br>
self.training_type_plugin.barrier(“pre_setup”)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py”, line 403, in barrier<br>
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 2776, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
Exception raised from ~AutoNcclGroup at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x14d6935a77d2 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const</em>, char const</em>, unsigned int, std::string const&amp;) + 0x5b (0x14d6935a3e6b in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span>  + 0x1145f2a (0x14d694bfff2a in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0x115106d (0x14d694c0b06d in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span> c10d::ProcessGroupNCCL::allreduce_impl(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0xf (0x14d694c0c09f in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span> c10d::ProcessGroupNCCL::allreduce(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0x2d3 (0x14d694c11ea3 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&amp;) + 0x72a (0x14d694c1b7ba in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0x8301e5 (0x14d6e6e131e5 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#8:</span>  + 0x1f6aa1 (0x14d6e67d9aa1 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#9:</span> _PyMethodDef_RawFastCallKeywords + 0x2ec (0x556d69f1d83c in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#10:</span> _PyObject_FastCallKeywords + 0x130 (0x556d69f53140 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#11:</span>  + 0x17fbd1 (0x556d69f53bd1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#12:</span> _PyEval_EvalFrameDefault + 0x1401 (0x556d69f983a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#13:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#14:</span> _PyFunction_FastCallKeywords + 0x583 (0x556d69f0ccd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#15:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#16:</span> _PyEval_EvalFrameDefault + 0x1401 (0x556d69f983a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#17:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#18:</span> _PyFunction_FastCallKeywords + 0x583 (0x556d69f0ccd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#19:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#20:</span> _PyEval_EvalFrameDefault + 0x661 (0x556d69f97601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#21:</span> _PyFunction_FastCallKeywords + 0x187 (0x556d69f0c8d7 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#22:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#23:</span> _PyEval_EvalFrameDefault + 0x661 (0x556d69f97601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#24:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#25:</span> _PyFunction_FastCallKeywords + 0x583 (0x556d69f0ccd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#26:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#27:</span> _PyEval_EvalFrameDefault + 0x1401 (0x556d69f983a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#28:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#29:</span> _PyObject_FastCallDict + 0x312 (0x556d69eee592 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#30:</span>  + 0x12f1c3 (0x556d69f031c3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#31:</span> PyObject_Call + 0xb4 (0x556d69eeeb94 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#32:</span> _PyEval_EvalFrameDefault + 0x1cb8 (0x556d69f98c58 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#33:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#34:</span> _PyFunction_FastCallKeywords + 0x583 (0x556d69f0ccd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#35:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#36:</span> _PyEval_EvalFrameDefault + 0x661 (0x556d69f97601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#37:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#38:</span> _PyFunction_FastCallKeywords + 0x521 (0x556d69f0cc71 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#39:</span>  + 0x17f9c5 (0x556d69f539c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#40:</span> _PyEval_EvalFrameDefault + 0x661 (0x556d69f97601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#41:</span> _PyEval_EvalCodeWithName + 0xdf9 (0x556d69eeda29 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#42:</span> _PyFunction_FastCallKeywords + 0x583 (0x556d69f0ccd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#43:</span> _PyEval_EvalFrameDefault + 0x3f5 (0x556d69f97395 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#44:</span> _PyFunction_FastCallKeywords + 0x187 (0x556d69f0c8d7 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#45:</span> _PyEval_EvalFrameDefault + 0x3f5 (0x556d69f97395 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#46:</span> _PyEval_EvalCodeWithName + 0x255 (0x556d69eece85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#47:</span> PyEval_EvalCode + 0x23 (0x556d69eee273 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#48:</span>  + 0x227c82 (0x556d69ffbc82 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#49:</span> PyRun_FileExFlags + 0x9e (0x556d6a005e1e in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#50:</span> PyRun_SimpleFileExFlags + 0x1bb (0x556d6a00600b in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#51:</span>  + 0x2330fa (0x556d6a0070fa in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#52:</span> _Py_UnixMain + 0x3c (0x556d6a00718c in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#53:</span> __libc_start_main + 0xf3 (0x14d6f61384a3 in /usr/lib64/libc.so.6)<br>
frame <span class="hashtag">#54:</span>  + 0x1d803a (0x556d69fac03a in /home1/anaconda3/envs/my_env/bin/python)</p>
<pre><code>trainer.fit(model, train_loader.train_dataloader(), val_loader.val_dataloader())
</code></pre>
<p>File “/home1/anaconda3/envs/cmy_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, <em><em>kwargs)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1138, in _run<br>
self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1435, in _call_setup_hook<br>
self.training_type_plugin.barrier(“pre_setup”)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py”, line 403, in barrier<br>
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 2776, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
Exception raised from ~AutoNcclGroup at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x1552110d17d2 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const</em>, char const</em>, unsigned int, std::string const&amp;) + 0x5b (0x1552110cde6b in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span>  + 0x1145f2a (0x155212729f2a in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0x115106d (0x15521273506d in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span> c10d::ProcessGroupNCCL::allreduce_impl(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0xf (0x15521273609f in /home1//anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span> c10d::ProcessGroupNCCL::allreduce(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0x2d3 (0x15521273bea3 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&amp;) + 0x72a (0x1552127457ba in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0x8301e5 (0x15526493d1e5 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#8:</span>  + 0x1f6aa1 (0x155264303aa1 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
<br>
frame <span class="hashtag">#53:</span> __libc_start_main + 0xf3 (0x155273c624a3 in /usr/lib64/libc.so.6)</p>

ptrblck: <p>Unfortunately, the NCCL logs are gone again. It might be easier if you could post a minimal, executable code snippet to reproduce the issue as well as the output of <code>python -m torch.utils.collect_env</code>, please?</p>

r00bi: <p>Collecting environment information…<br>
PyTorch version: 1.11.0+cu102<br>
Is debug build: False<br>
CUDA used to build PyTorch: 10.2<br>
ROCM used to build PyTorch: N/A</p>
<p>OS: Rocky Linux release 8.4 (Green Obsidian) (x86_64)<br>
GCC version: (GCC) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.21.3<br>
Libc version: glibc-2.17</p>
<p>Python version: 3.7.13 (default, Mar 29 2022, 02:18:16)  [GCC 7.5.0] (64-bit runtime)<br>
Python platform: Linux-4.18.0-305.12.1.el8_4.x86_64-x86_64-with-redhat-8.4-Green_Obsidian<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: NVIDIA A100-PCIE-40GB<br>
GPU 1: NVIDIA A100-PCIE-40GB</p>
<p>Nvidia driver version: 470.63.01<br>
cuDNN version: Could not collect<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.21.6<br>
[pip3] pytorch-lightning==1.5.10<br>
[pip3] torch==1.11.0<br>
[pip3] torchmetrics==0.9.1<br>
[conda] numpy                     1.21.6                   pypi_0    pypi<br>
[conda] pytorch-lightning         1.5.10                   pypi_0    pypi<br>
[conda] torch                     1.11.0                   pypi_0    pypi<br>
[conda] torchmetrics              0.9.1                    pypi_0    pypi</p>
<p>c305-001:221921:221921 [0] enqueue.cc:102 NCCL WARN Cuda failure ‘invalid device function’<br>
c305-001:221921:221921 [0] NCCL INFO Bootstrap : Using ib0:192.168.41.97&lt;0&gt;<br>
c305-001:221921:221921 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
c305-001:221921:221921 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:192.168.41.97&lt;0&gt;<br>
c305-001:221921:221921 [0] NCCL INFO Using network IB<br>
NCCL version 2.10.3+cuda10.2</p>
<p>c305-001:222031:222031 [1] enqueue.cc:102 NCCL WARN Cuda failure ‘invalid device function’<br>
c305-001:222031:222031 [1] NCCL INFO Bootstrap : Using ib0:192.168.41.97&lt;0&gt;<br>
c305-001:222031:222031 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
c305-001:222031:222031 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB ; OOB ib0:192.168.41.97&lt;0&gt;<br>
c305-001:222031:222031 [1] NCCL INFO Using network IB<br>
c305-001:222031:222058 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;0<br>
c305-001:221921:222056 [0] NCCL INFO Channel 00/02 :    0   1<br>
c305-001:222031:222058 [1] NCCL INFO Setting affinity for GPU 1 to ffffffff,ffffffff,00000000,00000000<br>
c305-001:221921:222056 [0] NCCL INFO Channel 01/02 :    0   1<br>
c305-001:221921:222056 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1<br>
c305-001:221921:222056 [0] NCCL INFO Setting affinity for GPU 0 to ffffffff,ffffffff<br>
c305-001:222031:222058 [1] NCCL INFO Channel 00 : 1[81000] → 0[21000] via P2P/IPC<br>
c305-001:222031:222058 [1] NCCL INFO Channel 01 : 1[81000] → 0[21000] via P2P/IPC<br>
c305-001:221921:222056 [0] NCCL INFO Channel 00 : 0[21000] → 1[81000] via P2P/IPC<br>
c305-001:221921:222056 [0] NCCL INFO Channel 01 : 0[21000] → 1[81000] via P2P/IPC<br>
c305-001:222031:222058 [1] NCCL INFO Connected all rings<br>
c305-001:222031:222058 [1] NCCL INFO Connected all trees<br>
c305-001:221921:222056 [0] NCCL INFO Connected all rings<br>
c305-001:221921:222056 [0] NCCL INFO Connected all trees<br>
c305-001:222031:222058 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
c305-001:222031:222058 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
c305-001:221921:222056 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/512<br>
c305-001:221921:222056 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
c305-001:222031:222058 [1] NCCL INFO comm 0x14646c001240 rank 1 nranks 2 cudaDev 1 busId 81000 - Init COMPLETE<br>
c305-001:221921:222056 [0] NCCL INFO comm 0x1546b8001240 rank 0 nranks 2 cudaDev 0 busId 21000 - Init COMPLETE<br>
c305-001:221921:221921 [0] NCCL INFO Launch mode Parallel</p>
<p>c305-001:221921:221921 [0] enqueue.cc:300 NCCL WARN Cuda failure ‘invalid device function’<br>
c305-001:221921:221921 [0] NCCL INFO group.cc:347 → 1</p>
<p>c305-001:222031:222031 [1] enqueue.cc:300 NCCL WARN Cuda failure ‘invalid device function’<br>
c305-001:222031:222031 [1] NCCL INFO group.cc:347 → 1</p>

r00bi: <h2>
<a name="initializing-distributed-global_rank-1-member-22-info-added-key-store_based_barrier_key1-to-store-for-rank-1-info-added-key-store_based_barrier_key1-to-store-for-rank-0-info-rank-0-completed-store-based-barrier-for-keystore_based_barrier_key1-with-2-nodes-1" class="anchor" href="#initializing-distributed-global_rank-1-member-22-info-added-key-store_based_barrier_key1-to-store-for-rank-1-info-added-key-store_based_barrier_key1-to-store-for-rank-0-info-rank-0-completed-store-based-barrier-for-keystore_based_barrier_key1-with-2-nodes-1"></a>initializing distributed: GLOBAL_RANK: 1, MEMBER: 2/2<br>
INFO: Added key: store_based_barrier_key:1 to store for rank: 1<br>
INFO: Added key: store_based_barrier_key:1 to store for rank: 0<br>
INFO: Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.</h2>
<h2>
<a name="distributed_backendnccl-all-distributed-processes-registered-starting-with-2-processes-2" class="anchor" href="#distributed_backendnccl-all-distributed-processes-registered-starting-with-2-processes-2"></a>distributed_backend=nccl<br>
All distributed processes registered. Starting with 2 processes</h2>
<p>INFO: Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.<br>
/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:145: UserWarning:<br>
NVIDIA A100-PCIE-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.<br>
If you want to use the NVIDIA A100-PCIE-40GB GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))<br>
/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py:510: UserWarning: Error handling mechanism for deadlock detection is uninitialized. Skipping check.<br>
rank_zero_warn(“Error handling mechanism for deadlock detection is uninitialized. Skipping check.”)<br>
/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/cuda/<strong>init</strong>.py:145: UserWarning:<br>
NVIDIA A100-PCIE-40GB with CUDA capability sm_80 is not compatible with the current PyTorch installation.<br>
The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.<br>
If you want to use the NVIDIA A100-PCIE-40GB GPU with PyTorch, please check the instructions at <a href="https://pytorch.org/get-started/locally/" class="inline-onebox" rel="noopener nofollow ugc">Start Locally | PyTorch</a></p>
<p>warnings.warn(incompatible_device_warn.format(device_name, capability, " ".join(arch_list), device_name))<br>
Traceback (most recent call last):<br>
File “~/novo/novo.py”, line 63, in <br>
main()<br>
File “~/novo/novo.py”, line 47, in main<br>
train(train_data_path, val_data_path, model_path, config_path)<br>
File “/work/08447/se0204/Transformer/main/novo/denovo/train_test.py”, line 144, in train<br>
trainer.fit(model, train_loader.train_dataloader(), val_loader.val_dataloader())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, <em><em>kwargs)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1138, in _run<br>
self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1435, in _call_setup_hook<br>
self.training_type_plugin.barrier(“pre_setup”)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py”, line 403, in barrier<br>
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 2776, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
Exception raised from ~AutoNcclGroup at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x1464f1e5e7d2 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const</em>, char const</em>, unsigned int, std::string const&amp;) + 0x5b (0x1464f1e5ae6b in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span>  + 0x1145f2a (0x1464f34b6f2a in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0x115106d (0x1464f34c206d in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span> c10d::ProcessGroupNCCL::allreduce_impl(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0xf (0x1464f34c309f in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span> c10d::ProcessGroupNCCL::allreduce(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0x2d3 (0x1464f34c8ea3 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&amp;) + 0x72a (0x1464f34d27ba in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0x8301e5 (0x1465456ca1e5 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#8:</span>  + 0x1f6aa1 (0x146545090aa1 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#9:</span> _PyMethodDef_RawFastCallKeywords + 0x2ec (0x55ca311e283c in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#10:</span> _PyObject_FastCallKeywords + 0x130 (0x55ca31218140 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#11:</span>  + 0x17fbd1 (0x55ca31218bd1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#12:</span> _PyEval_EvalFrameDefault + 0x1401 (0x55ca3125d3a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#13:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#14:</span> _PyFunction_FastCallKeywords + 0x583 (0x55ca311d1cd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#15:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#16:</span> _PyEval_EvalFrameDefault + 0x1401 (0x55ca3125d3a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#17:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#18:</span> _PyFunction_FastCallKeywords + 0x583 (0x55ca311d1cd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#19:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#20:</span> _PyEval_EvalFrameDefault + 0x661 (0x55ca3125c601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#21:</span> _PyFunction_FastCallKeywords + 0x187 (0x55ca311d18d7 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#22:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#23:</span> _PyEval_EvalFrameDefault + 0x661 (0x55ca3125c601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#24:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#25:</span> _PyFunction_FastCallKeywords + 0x583 (0x55ca311d1cd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#26:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#27:</span> _PyEval_EvalFrameDefault + 0x1401 (0x55ca3125d3a1 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#28:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#29:</span> _PyObject_FastCallDict + 0x312 (0x55ca311b3592 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#30:</span>  + 0x12f1c3 (0x55ca311c81c3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#31:</span> PyObject_Call + 0xb4 (0x55ca311b3b94 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#32:</span> _PyEval_EvalFrameDefault + 0x1cb8 (0x55ca3125dc58 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#33:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#34:</span> _PyFunction_FastCallKeywords + 0x583 (0x55ca311d1cd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#35:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#36:</span> _PyEval_EvalFrameDefault + 0x661 (0x55ca3125c601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#37:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#38:</span> _PyFunction_FastCallKeywords + 0x521 (0x55ca311d1c71 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#39:</span>  + 0x17f9c5 (0x55ca312189c5 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#40:</span> _PyEval_EvalFrameDefault + 0x661 (0x55ca3125c601 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#41:</span> _PyEval_EvalCodeWithName + 0xdf9 (0x55ca311b2a29 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#42:</span> _PyFunction_FastCallKeywords + 0x583 (0x55ca311d1cd3 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#43:</span> _PyEval_EvalFrameDefault + 0x3f5 (0x55ca3125c395 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#44:</span> _PyFunction_FastCallKeywords + 0x187 (0x55ca311d18d7 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#45:</span> _PyEval_EvalFrameDefault + 0x3f5 (0x55ca3125c395 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#46:</span> _PyEval_EvalCodeWithName + 0x255 (0x55ca311b1e85 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#47:</span> PyEval_EvalCode + 0x23 (0x55ca311b3273 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#48:</span>  + 0x227c82 (0x55ca312c0c82 in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#49:</span> PyRun_FileExFlags + 0x9e (0x55ca312cae1e in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#50:</span> PyRun_SimpleFileExFlags + 0x1bb (0x55ca312cb00b in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#51:</span>  + 0x2330fa (0x55ca312cc0fa in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#52:</span> _Py_UnixMain + 0x3c (0x55ca312cc18c in /home1/anaconda3/envs/my_env/bin/python)<br>
frame <span class="hashtag">#53:</span> __libc_start_main + 0xf3 (0x1465549ef4a3 in /usr/lib64/libc.so.6)<br>
frame <span class="hashtag">#54:</span>  + 0x1d803a (0x55ca3127103a in /home1/anaconda3/envs/my_env/bin/python)</p>
<p>Traceback (most recent call last):<br>
File “/work/08447/se0204/Transformer/main/novo/novo.py”, line 63, in <br>
main()<br>
File “~/novo.py”, line 47, in main<br>
train(train_data_path, val_data_path, model_path, config_path)<br>
File “~/novo/denovo/train_test.py”, line 144, in train<br>
trainer.fit(model, train_loader.train_dataloader(), val_loader.val_dataloader())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 741, in fit<br>
self._fit_impl, model, train_dataloaders, val_dataloaders, datamodule, ckpt_path<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 685, in _call_and_handle_interrupt<br>
return trainer_fn(*args, <em><em>kwargs)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 777, in _fit_impl<br>
self._run(model, ckpt_path=ckpt_path)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1138, in _run<br>
self._call_setup_hook()  # allow user to setup lightning_module in accelerator environment<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/trainer/trainer.py”, line 1435, in _call_setup_hook<br>
self.training_type_plugin.barrier(“pre_setup”)<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/pytorch_lightning/plugins/training_type/ddp.py”, line 403, in barrier<br>
torch.distributed.barrier(device_ids=self.determine_ddp_device_ids())<br>
File “/home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py”, line 2776, in barrier<br>
work = default_pg.barrier(opts=opts)<br>
RuntimeError: NCCL error in: …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47, unhandled cuda error, NCCL version 21.0.3<br>
ncclUnhandledCudaError: Call to CUDA function failed.<br>
Exception raised from ~AutoNcclGroup at …/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:47 (most recent call first):<br>
frame <span class="hashtag">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x1547458d27d2 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#1:</span> c10::detail::torchCheckFail(char const</em>, char const</em>, unsigned int, std::string const&amp;) + 0x5b (0x1547458cee6b in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag">#2:</span>  + 0x1145f2a (0x154746f2af2a in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#3:</span>  + 0x115106d (0x154746f3606d in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#4:</span> c10d::ProcessGroupNCCL::allreduce_impl(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0xf (0x154746f3709f in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#5:</span> c10d::ProcessGroupNCCL::allreduce(std::vector&lt;at::Tensor, std::allocator<a>at::Tensor</a> &gt;&amp;, c10d::AllreduceOptions const&amp;) + 0x2d3 (0x154746f3cea3 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#6:</span> c10d::ProcessGroupNCCL::barrier(c10d::BarrierOptions const&amp;) + 0x72a (0x154746f467ba in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_cuda.so)<br>
frame <span class="hashtag">#7:</span>  + 0x8301e5 (0x15479913e1e5 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag">#8:</span>  + 0x1f6aa1 (0x154798b04aa1 in /home1/anaconda3/envs/my_env/lib/python3.7/site-packages/torch/lib/libtorch_python.so)<br>
<br>
frame <span class="hashtag">#53:</span> __libc_start_main + 0xf3 (0x1547a84634a3 in /usr/lib64/libc.so.6)</p>

ptrblck: <p>Thanks for the updated logs.<br>
The error is raised by:</p>
<pre><code class="lang-python">The current PyTorch install supports CUDA capabilities sm_37 sm_50 sm_60 sm_70.
If you want to use the NVIDIA A100-PCIE-40GB GPU with PyTorch, please check the instructions at Start Locally | PyTorch
</code></pre>
<p>and NCCL just reraises it.<br>
Your A100 GPU needs CUDA&gt;=11.0 so install the binaries with the 11.3 or 11.6 runtime.</p>

Niraj_Bhujel: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am stuck for last few days on this error below when running the script using torch.distributed.launch <code>python3.8 -m torch.distributed.launch \     --nnodes=1 \     --node_rank=0 \     --nproc_per_node=2 \     train_stdec.py</code>.</p>
<p>Sorry for my inability to post question properly. The error looks like this.</p>
<blockquote>
<p>deep_lab:31220:31363 [1] enqueue.cc:215 NCCL WARN Cuda failure ‘the launch timed out and was terminated’<br>
deep_lab:31220:31363 [1] NCCL INFO group.cc:282 → 1<br>
Traceback (most recent call last):<br>
File “train_stdec.py”, line 743, in main<br>
run(args, model_params, device, prediction_model)<br>
File “train_stdec.py”, line 493, in run<br>
h = train_epoch(args, model, prediction_model, optimizer, dataloaders[‘train’], device, epoch, history=h)<br>
File “train_stdec.py”, line 368, in train_epoch<br>
epoch_loss.backward(retain_graph=False)<br>
File “/home/deep_lab/.local/lib/python3.8/site-packages/torch/tensor.py”, line 245, in backward<br>
torch.autograd.backward(self, gradient, retain_graph, create_graph, inputs=inputs)<br>
File “/home/deep_lab/.local/lib/python3.8/site-packages/torch/autograd/<strong>init</strong>.py”, line 145, in backward<br>
Variable._execution_engine.run_backward(<br>
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:33, unhandled cuda error, NCCL version 2.7.8<br>
ncclUnhandledCudaError: Call to CUDA function failed.</p>
</blockquote>
<p>Here is the output from NCCL_DEBUG=INFO</p>
<blockquote>
<p>deep_lab:31220:31220 [1] NCCL INFO Bootstrap : Using [0]enp11s0:10.2.3.219&lt;0&gt;<br>
deep_lab:31220:31220 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation<br>
deep_lab:31220:31220 [1] NCCL INFO NET/IB : No device found.<br>
deep_lab:31220:31220 [1] NCCL INFO NET/Socket : Using [0]enp11s0:10.2.3.219&lt;0&gt;<br>
deep_lab:31220:31220 [1] NCCL INFO Using network Socket<br>
deep_lab:31219:31300 [0] NCCL INFO Channel 00/02 :    0   1<br>
deep_lab:31220:31303 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64<br>
deep_lab:31219:31300 [0] NCCL INFO Channel 01/02 :    0   1<br>
deep_lab:31220:31303 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;-1/-1/-1 [1] -1/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;-1/-1/-1<br>
deep_lab:31219:31300 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 8/8/64<br>
deep_lab:31219:31300 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [1] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1<br>
deep_lab:31219:31300 [0] NCCL INFO Channel 00 : 0[1000] → 1[2000] via direct shared memory<br>
deep_lab:31220:31303 [1] NCCL INFO Channel 00 : 1[2000] → 0[1000] via direct shared memory<br>
deep_lab:31219:31300 [0] NCCL INFO Channel 01 : 0[1000] → 1[2000] via direct shared memory<br>
deep_lab:31220:31303 [1] NCCL INFO Channel 01 : 1[2000] → 0[1000] via direct shared memory<br>
deep_lab:31219:31300 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
deep_lab:31219:31300 [0] NCCL INFO comm 0x7f33c0002e10 rank 0 nranks 2 cudaDev 0 busId 1000 - Init COMPLETE<br>
deep_lab:31219:31219 [0] NCCL INFO Launch mode Parallel<br>
deep_lab:31220:31303 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer<br>
deep_lab:31220:31303 [1] NCCL INFO comm 0x7f2150002e10 rank 1 nranks 2 cudaDev 1 busId 2000 - Init COMPLETE</p>
</blockquote>
<p>The output from CUDA_LAUNCH_BLOCKING=1 is as below:</p>
<blockquote>
<p>terminate called after throwing an instance of ‘c10::Error’<br>
what():  CUDA error: the launch timed out and was terminated<br>
Exception raised from create_event_internal at /pytorch/c10/cuda/CUDACachingAllocator.cpp:733 (most recent call first):<br>
frame <span class="hashtag-raw">#0:</span> c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7f226f5a92f2 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#1:</span> c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::string const&amp;) + 0x5b (0x7f226f5a667b in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#2:</span> c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x809 (0x7f226f8011f9 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libc10_cuda.so)<br>
frame <span class="hashtag-raw">#3:</span> c10::TensorImpl::release_resources() + 0x54 (0x7f226f5913a4 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libc10.so)<br>
frame <span class="hashtag-raw">#4:</span> std::vector&lt;c10d::Reducer::Bucket, std::allocator<a>c10d::Reducer::Bucket</a> &gt;::~vector() + 0x2f9 (0x7f22e36d5cc9 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#5:</span> c10d::Reducer::~Reducer() + 0x26a (0x7f22e36cac8a in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#6:</span> std::_Sp_counted_ptr&lt;c10d::Reducer*, (__gnu_cxx::_Lock_policy)2&gt;::_M_dispose() + 0x12 (0x7f22e36f1f22 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#7:</span> std::_Sp_counted_base&lt;(__gnu_cxx::_Lock_policy)2&gt;::_M_release() + 0x46 (0x7f22e302ee76 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#8:</span>  + 0xa2121f (0x7f22e36f521f in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#9:</span>  + 0x369f80 (0x7f22e303df80 in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#10:</span>  + 0x36b1ee (0x7f22e303f1ee in /home/deep_lab/.local/lib/python3.8/site-packages/torch/lib/libtorch_python.so)<br>
frame <span class="hashtag-raw">#11:</span> /usr/bin/python3.8() [0x5d28f4]<br>
frame <span class="hashtag-raw">#12:</span> /usr/bin/python3.8() [0x5a729d]<br>
frame <span class="hashtag-raw">#13:</span> /usr/bin/python3.8() [0x5ec780]<br>
frame <span class="hashtag-raw">#14:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#15:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#16:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#17:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#18:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#19:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#20:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#21:</span> /usr/bin/python3.8() [0x5ec84a]<br>
frame <span class="hashtag-raw">#22:</span> /usr/bin/python3.8() [0x5441f8]<br>
frame <span class="hashtag-raw">#23:</span> /usr/bin/python3.8() [0x5ef9e6]<br>
frame <span class="hashtag-raw">#24:</span> /usr/bin/python3.8() [0x6af687]<br>
frame <span class="hashtag-raw">#25:</span> /usr/bin/python3.8() [0x5ef9df]<br>
frame <span class="hashtag-raw">#26:</span> /usr/bin/python3.8() [0x6af687]<br>
frame <span class="hashtag-raw">#27:</span> _PyModule_ClearDict + 0xe0c (0x5c6aac in /usr/bin/python3.8)<br>
frame <span class="hashtag-raw">#28:</span> PyImport_Cleanup + 0x2be (0x68485e in /usr/bin/python3.8)<br>
frame <span class="hashtag-raw">#29:</span> Py_FinalizeEx + 0x7f (0x67f8af in /usr/bin/python3.8)<br>
frame <span class="hashtag-raw">#30:</span> Py_RunMain + 0x32d (0x6b70fd in /usr/bin/python3.8)<br>
frame <span class="hashtag-raw">#31:</span> Py_BytesMain + 0x2d (0x6b736d in /usr/bin/python3.8)<br>
frame <span class="hashtag-raw">#32:</span> __libc_start_main + 0xf3 (0x7f22e9216083 in /lib/x86_64-linux-gnu/libc.so.6)<br>
frame <span class="hashtag-raw">#33:</span> _start + 0x2e (0x5fa5ce in /usr/bin/python3.8)</p>
</blockquote>
<p>The output from  python -m torch.utils.collect_env looks like this:</p>
<blockquote>
<p>Collecting environment information…<br>
PyTorch version: 1.8.1+cu111<br>
Is debug build: False<br>
CUDA used to build PyTorch: 11.1<br>
ROCM used to build PyTorch: N/A</p>
</blockquote>
<blockquote>
<p>OS: Ubuntu 20.04.4 LTS (x86_64)<br>
GCC version: (Ubuntu 9.4.0-1ubuntu1~20.04.1) 9.4.0<br>
Clang version: Could not collect<br>
CMake version: version 3.16.3</p>
</blockquote>
<blockquote>
<p>Python version: 3.8 (64-bit runtime)<br>
Is CUDA available: True<br>
CUDA runtime version: Could not collect<br>
GPU models and configuration:<br>
GPU 0: NVIDIA GeForce GTX TITAN X<br>
GPU 1: NVIDIA GeForce GTX TITAN X</p>
</blockquote>
<blockquote>
<p>Nvidia driver version: 465.19.01<br>
cuDNN version: Probably one of the following:<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.5<br>
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.4.1<br>
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.4.1<br>
HIP runtime version: N/A<br>
MIOpen runtime version: N/A</p>
</blockquote>
<blockquote>
<p>Versions of relevant libraries:<br>
[pip3] numpy==1.23.0<br>
[pip3] numpydoc==1.4.0<br>
[pip3] torch==1.8.1+cu111<br>
[pip3] torchaudio==0.8.1<br>
[pip3] torchvision==0.9.1+cu111<br>
[conda] Could not collect</p>
</blockquote>

ptrblck: <p>This error:</p>
<pre><code class="lang-python"> CUDA error: the launch timed out and was terminated
</code></pre>
<p>is raised if the OS kills the kernel.<br>
This is usually the case if the GPU is used for video output (e.g. X window) and the OS wants to avoid a lagging video output.<br>
As a workaround you could kill the X server and rerun the workload in a terminal only to see if it would be working.</p>

Niraj_Bhujel: <p>HI <a class="mention" href="/u/ptrblck">@ptrblck</a> , thank you for your kind reply. I confirm that distributed training is working when X server is killed. However, I lost the video output as you said. As a workaround, I commented all the lines in <code>/usr/share/X11/xorg.conf.d/10-nvidia.conf</code> following <a href="https://askubuntu.com/a/1313440" rel="noopener nofollow ugc">https://askubuntu.com/a/1313440</a> and reboot my PC. The distributed training still works, however, the Nvidia X server settings window will be empty.</p>

user27: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a> , I am stuck on following error for 1 week. Can you please help fix this issue:</p>
<pre><code class="lang-auto">enqueue.cc:115 NCCL WARN Cuda failure 'named symbol not found'

user-0:2395:2560 [0] enqueue.cc:128 NCCL WARN Cuda failure 'named symbol not found'
user-0:2395:2560 [0] NCCL INFO init.cc:1369 -&gt; 1
user-0:2395:2560 [0] NCCL INFO group.cc:65 -&gt; 1 [Async thread]
user-0:2395:2395 [0] NCCL INFO group.cc:406 -&gt; 1
user-0:2395:2395 [0] NCCL INFO group.cc:96 -&gt; 1
2024-03-01 10:04:51,604 [INFO] - s3_checkpoint_io.py:200 - Entering teardown, waiting for all jobs to finish, rank 0
2024-03-01 10:04:51,617 [INFO] - s3_checkpoint_io.py:203 - executor shut down after 0.01 seconds, rank 0
Error executing job with overrides: ['user=user' ]
  File "/train.py", line 312, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in &lt;lambda&gt;
    lambda: hydra.run(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/train.py", line 268, in main
    trainer.fit(model=model_module, datamodule=data_module)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 939, in _run
    self.__setup_profiler()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1069, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1192, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/ddp.py", line 292, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2716, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2019, in broadcast
    work = default_pg.broadcast([tensor], opts)
Traceback (most recent call last):
  File "/train.py", line 312, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in &lt;lambda&gt;
    lambda: hydra.run(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/train.py", line 268, in main
    trainer.fit(model=model_module, datamodule=data_module)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 939, in _run
    self.__setup_profiler()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1069, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1192, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/ddp.py", line 292, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2716, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2019, in broadcast
    work = default_pg.broadcast([tensor], opts)
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'named symbol not found'
Exception raised from getNCCLComm at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0xae (0x7fd5895b14ee in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xbacd1a (0x7fd532c67d1a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::broadcast(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;, c10d::BroadcastOptions const&amp;) + 0x4a8 (0x7fd532f08398 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0x4e9c7a0 (0x7fd57fbc97a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x4eae2e3 (0x7fd57fbdb2e3 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: &lt;unknown function&gt; + 0x4eb7811 (0x7fd57fbe4811 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x4ec8971 (0x7fd57fbf5971 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #7: &lt;unknown function&gt; + 0xc459a5 (0x7fd5871439a5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #8: &lt;unknown function&gt; + 0x3c47cf (0x7fd5868c27cf in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;

Error in training on pod user-0, ip address, instance id i-07b659df0fac77efe.  Exception: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'named symbol not found'
Exception raised from getNCCLComm at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0xae (0x7fd5895b14ee in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xbacd1a (0x7fd532c67d1a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::broadcast(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;, c10d::BroadcastOptions const&amp;) + 0x4a8 (0x7fd532f08398 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0x4e9c7a0 (0x7fd57fbc97a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x4eae2e3 (0x7fd57fbdb2e3 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: &lt;unknown function&gt; + 0x4eb7811 (0x7fd57fbe4811 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x4ec8971 (0x7fd57fbf5971 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #7: &lt;unknown function&gt; + 0xc459a5 (0x7fd5871439a5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #8: &lt;unknown function&gt; + 0x3c47cf (0x7fd5868c27cf in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;

  File "/train.py", line 312, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in &lt;lambda&gt;
    lambda: hydra.run(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/train.py", line 268, in main
    trainer.fit(model=model_module, datamodule=data_module)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 939, in _run
    self.__setup_profiler()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1069, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1192, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/ddp.py", line 292, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2716, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2019, in broadcast
    work = default_pg.broadcast([tensor], opts)
Traceback (most recent call last):
  File "/train.py", line 329, in &lt;module&gt;
    raise e
  File "/train.py", line 312, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.10/dist-packages/hydra/main.py", line 94, in decorated_main
    _run_hydra(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 223, in run_and_report
    raise ex
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 220, in run_and_report
    return func()
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/utils.py", line 458, in &lt;lambda&gt;
    lambda: hydra.run(
  File "/usr/local/lib/python3.10/dist-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/usr/local/lib/python3.10/dist-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/train.py", line 268, in main
    trainer.fit(model=model_module, datamodule=data_module)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 532, in fit
    call._call_and_handle_interrupt(
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/call.py", line 42, in _call_and_handle_interrupt
    return trainer.strategy.launcher.launch(trainer_fn, *args, trainer=trainer, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/launchers/subprocess_script.py", line 93, in launch
    return function(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 571, in _fit_impl
    self._run(model, ckpt_path=ckpt_path)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 939, in _run
    self.__setup_profiler()
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1069, in __setup_profiler
    self.profiler.setup(stage=self.state.fn, local_rank=local_rank, log_dir=self.log_dir)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/trainer/trainer.py", line 1192, in log_dir
    dirpath = self.strategy.broadcast(dirpath)
  File "/usr/local/lib/python3.10/dist-packages/pytorch_lightning/strategies/ddp.py", line 292, in broadcast
    torch.distributed.broadcast_object_list(obj, src, group=_group.WORLD)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2716, in broadcast_object_list
    broadcast(object_sizes_tensor, src=src, group=group)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py", line 2019, in broadcast
    work = default_pg.broadcast([tensor], opts)
torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5
ncclUnhandledCudaError: Call to CUDA function failed.
Last error:
Cuda failure 'named symbol not found'
Exception raised from getNCCLComm at /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1359 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;) + 0xae (0x7fd5895b14ee in /usr/local/lib/python3.10/dist-packages/torch/lib/libc10.so)
frame #1: &lt;unknown function&gt; + 0xbacd1a (0x7fd532c67d1a in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #2: c10d::ProcessGroupNCCL::broadcast(std::vector&lt;at::Tensor, std::allocator&lt;at::Tensor&gt; &gt;&amp;, c10d::BroadcastOptions const&amp;) + 0x4a8 (0x7fd532f08398 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cuda.so)
frame #3: &lt;unknown function&gt; + 0x4e9c7a0 (0x7fd57fbc97a0 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #4: &lt;unknown function&gt; + 0x4eae2e3 (0x7fd57fbdb2e3 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #5: &lt;unknown function&gt; + 0x4eb7811 (0x7fd57fbe4811 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #6: &lt;unknown function&gt; + 0x4ec8971 (0x7fd57fbf5971 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_cpu.so)
frame #7: &lt;unknown function&gt; + 0xc459a5 (0x7fd5871439a5 in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
frame #8: &lt;unknown function&gt; + 0x3c47cf (0x7fd5868c27cf in /usr/local/lib/python3.10/dist-packages/torch/lib/libtorch_python.so)
&lt;omitting python frames&gt;

user-0:2395:2395 [0] NCCL INFO comm 0x55f5f56e42d0 rank 0 nranks 0 cudaDev 0 busId 0 - Abort COMPLETE

</code></pre>
<p>Appreciate your help on this!</p>