Error when training with DataParallel

username: Thien_Tan
cooked: <p>I train my model with DDP with command: NCCL_DEBUG=INFO /bin/python3 -m torch.distributed.launch --nproc_per_node=2 --master_port=1234 /home/toan/HAT_Model/test.py</p>
<p>It throw this error:<br>
warnings.warn(<br>
[2024-08-13 15:44:52,787] torch.distributed.run: [WARNING]<br>
[2024-08-13 15:44:52,787] torch.distributed.run: [WARNING] *****************************************<br>
[2024-08-13 15:44:52,787] torch.distributed.run: [WARNING] Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed.<br>
[2024-08-13 15:44:52,787] torch.distributed.run: [WARNING] *****************************************<br>
/home/toan/.local/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at …/aten/src/ATen/native/TensorShape.cpp:3526.)<br>
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]<br>
/home/toan/.local/lib/python3.10/site-packages/torch/functional.py:504: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at …/aten/src/ATen/native/TensorShape.cpp:3526.)<br>
return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO cudaDriverVersion 12020<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO cudaDriverVersion 12020</p>
<p>DESKTOP-7DHJIHI:82116:82116 [0] misc/cudawrap.cc:33 NCCL WARN Cuda failure ‘initialization error’</p>
<p>DESKTOP-7DHJIHI:82115:82115 [0] misc/cudawrap.cc:33 NCCL WARN Cuda failure ‘initialization error’<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO Bootstrap : Using eth0:172.20.10.6&lt;0&gt;<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO Bootstrap : Using eth0:172.20.10.6&lt;0&gt;<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation<br>
NCCL version 2.18.1+cuda12.1NCCL version 2.18.1+cuda12.1</p>
<p>DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Failed to open libibverbs.so[.1]<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO NET/Socket : Using [0]eth0:172.20.10.6&lt;0&gt;<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Using network Socket<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Failed to open libibverbs.so[.1]<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO NET/Socket : Using [0]eth0:172.20.10.6&lt;0&gt;<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Using network Socket<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Using network Socket<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Using network Socket<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 00/04 :    0   1<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 01/04 :    0   1<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 02/04 :    0   1<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 03/04 :    0   1<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] -1/-1/-1-&gt;0-&gt;1 [2] 1/-1/-1-&gt;0-&gt;-1 [3] -1/-1/-1-&gt;0-&gt;1<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO P2P Chunksize set to 524288<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1 [2] -1/-1/-1-&gt;1-&gt;0 [3] 0/-1/-1-&gt;1-&gt;-1<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO P2P Chunksize set to 524288<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 00/04 :    0   1<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 01/04 :    0   1<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 02/04 :    0   1<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 03/04 :    0   1<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] 0/-1/-1-&gt;1-&gt;-1 [2] -1/-1/-1-&gt;1-&gt;0 [3] 0/-1/-1-&gt;1-&gt;-1<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO P2P Chunksize set to 524288<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] -1/-1/-1-&gt;0-&gt;1 [2] 1/-1/-1-&gt;0-&gt;-1 [3] -1/-1/-1-&gt;0-&gt;1<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO P2P Chunksize set to 524288<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Channel 00/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 00/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 00/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Channel 00/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Channel 01/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 01/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Channel 01/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 01/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Channel 02/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 02/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 02/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Channel 02/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Channel 03/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Channel 03/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Channel 03/0 : 0[2000] → 1[81000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Channel 03/0 : 1[81000] → 0[2000] via P2P/direct pointer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Connected all rings<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO Connected all trees<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO 4 coll channels, 0 nvls channels, 4 p2p channels, 4 p2p channels per peer<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Connected all rings<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO Connected all trees<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO 4 coll channels, 0 nvls channels, 4 p2p channels, 4 p2p channels per peer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Connected all rings<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Connected all rings<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO Connected all trees<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO Connected all trees<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO 4 coll channels, 0 nvls channels, 4 p2p channels, 4 p2p channels per peer<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO 4 coll channels, 0 nvls channels, 4 p2p channels, 4 p2p channels per peer<br>
DESKTOP-7DHJIHI:82115:82198 [0] NCCL INFO comm 0x55be8d2e1ce0 rank 0 nranks 2 cudaDev 0 busId 2000 commId 0x9bc561efd9a2224a - Init COMPLETE<br>
DESKTOP-7DHJIHI:82115:82199 [1] NCCL INFO comm 0x55be8d2e6a60 rank 1 nranks 2 cudaDev 1 busId 81000 commId 0x9bc561efd9a2224a - Init COMPLETE</p>
<p>DESKTOP-7DHJIHI:82115:82115 [0] misc/strongstream.cc:60 NCCL WARN Cuda failure ‘unknown error’<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO enqueue.cc:1550 → 1<br>
DESKTOP-7DHJIHI:82115:82115 [0] NCCL INFO enqueue.cc:1591 → 1<br>
DESKTOP-7DHJIHI:82115:82115 [1] NCCL INFO group.cc:106 → 1<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details)<br>
DESKTOP-7DHJIHI:82116:82201 [1] NCCL INFO comm 0x55b45e006740 rank 1 nranks 2 cudaDev 1 busId 81000 commId 0xcbe944600c283a75 - Init COMPLETE<br>
DESKTOP-7DHJIHI:82116:82200 [0] NCCL INFO comm 0x55b45e0019c0 rank 0 nranks 2 cudaDev 0 busId 2000 commId 0xcbe944600c283a75 - Init COMPLETE</p>
<h1><a name="desktop-7dhjihi8211682116-0-miscstrongstreamcc60-nccl-warn-cuda-failure-an-illegal-memory-access-was-encountered-desktop-7dhjihi8211682116-0-nccl-info-enqueuecc1550-1-desktop-7dhjihi8211682116-0-nccl-info-enqueuecc1591-1-desktop-7dhjihi8211682116-1-nccl-info-groupcc106-1-terminate-called-after-throwing-an-instance-of-stdruntime_error-what-nccl-error-1-unhandled-cuda-error-run-with-nccl_debuginfo-for-details-2024-08-13-154512817-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-6-local_rank-0-pid-82115-of-binary-binpython3-traceback-most-recent-call-last-file-usrlibpython310runpypy-line-196-in-_run_module_as_main-return-_run_codecode-main_globals-none-file-usrlibpython310runpypy-line-86-in-_run_code-execcode-run_globals-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-196-in-module-main-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-192-in-main-launchargs-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-177-in-launch-runargs-file-hometoanlocallibpython310site-packagestorchdistributedrunpy-line-797-in-run-elastic_launch-file-hometoanlocallibpython310site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-hometoanlocallibpython310site-packagestorchdistributedlauncherapipy-line-264-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1" class="anchor" href="#desktop-7dhjihi8211682116-0-miscstrongstreamcc60-nccl-warn-cuda-failure-an-illegal-memory-access-was-encountered-desktop-7dhjihi8211682116-0-nccl-info-enqueuecc1550-1-desktop-7dhjihi8211682116-0-nccl-info-enqueuecc1591-1-desktop-7dhjihi8211682116-1-nccl-info-groupcc106-1-terminate-called-after-throwing-an-instance-of-stdruntime_error-what-nccl-error-1-unhandled-cuda-error-run-with-nccl_debuginfo-for-details-2024-08-13-154512817-torchdistributedelasticmultiprocessingapi-error-failed-exitcode-6-local_rank-0-pid-82115-of-binary-binpython3-traceback-most-recent-call-last-file-usrlibpython310runpypy-line-196-in-_run_module_as_main-return-_run_codecode-main_globals-none-file-usrlibpython310runpypy-line-86-in-_run_code-execcode-run_globals-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-196-in-module-main-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-192-in-main-launchargs-file-hometoanlocallibpython310site-packagestorchdistributedlaunchpy-line-177-in-launch-runargs-file-hometoanlocallibpython310site-packagestorchdistributedrunpy-line-797-in-run-elastic_launch-file-hometoanlocallibpython310site-packagestorchdistributedlauncherapipy-line-134-in-__call__-return-launch_agentself_config-self_entrypoint-listargs-file-hometoanlocallibpython310site-packagestorchdistributedlauncherapipy-line-264-in-launch_agent-raise-childfailederror-torchdistributedelasticmultiprocessingerrorschildfailederror-1"></a>DESKTOP-7DHJIHI:82116:82116 [0] misc/strongstream.cc:60 NCCL WARN Cuda failure ‘an illegal memory access was encountered’<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO enqueue.cc:1550 → 1<br>
DESKTOP-7DHJIHI:82116:82116 [0] NCCL INFO enqueue.cc:1591 → 1<br>
DESKTOP-7DHJIHI:82116:82116 [1] NCCL INFO group.cc:106 → 1<br>
terminate called after throwing an instance of ‘std::runtime_error’<br>
what():  NCCL Error 1: unhandled cuda error (run with NCCL_DEBUG=INFO for details)<br>
[2024-08-13 15:45:12,817] torch.distributed.elastic.multiprocessing.api: [ERROR] failed (exitcode: -6) local_rank: 0 (pid: 82115) of binary: /bin/python3<br>
Traceback (most recent call last):<br>
File “/usr/lib/python3.10/runpy.py”, line 196, in _run_module_as_main<br>
return _run_code(code, main_globals, None,<br>
File “/usr/lib/python3.10/runpy.py”, line 86, in _run_code<br>
exec(code, run_globals)<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/launch.py”, line 196, in <br>
main()<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/launch.py”, line 192, in main<br>
launch(args)<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/launch.py”, line 177, in launch<br>
run(args)<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/run.py”, line 797, in run<br>
elastic_launch(<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 134, in <strong>call</strong><br>
return launch_agent(self._config, self._entrypoint, list(args))<br>
File “/home/toan/.local/lib/python3.10/site-packages/torch/distributed/launcher/api.py”, line 264, in launch_agent<br>
raise ChildFailedError(<br>
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:</h1>
<h2><a name="hometoanhat_modeltestpy-failed-2" class="anchor" href="#hometoanhat_modeltestpy-failed-2"></a>/home/toan/HAT_Model/test.py FAILED</h2>
<h2><a name="failures-1-time-2024-08-13_154512-host-desktop-7dhjihi-rank-1-local_rank-1-exitcode-6-pid-82116-error_file-na-traceback-signal-6-sigabrt-received-by-pid-82116-3" class="anchor" href="#failures-1-time-2024-08-13_154512-host-desktop-7dhjihi-rank-1-local_rank-1-exitcode-6-pid-82116-error_file-na-traceback-signal-6-sigabrt-received-by-pid-82116-3"></a>Failures:<br>
[1]:<br>
time      : 2024-08-13_15:45:12<br>
host      : DESKTOP-7DHJIHI.<br>
rank      : 1 (local_rank: 1)<br>
exitcode  : -6 (pid: 82116)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 82116</h2>
<h1><a name="root-cause-first-observed-failure-0-time-2024-08-13_154512-host-desktop-7dhjihi-rank-0-local_rank-0-exitcode-6-pid-82115-error_file-na-traceback-signal-6-sigabrt-received-by-pid-82115-4" class="anchor" href="#root-cause-first-observed-failure-0-time-2024-08-13_154512-host-desktop-7dhjihi-rank-0-local_rank-0-exitcode-6-pid-82115-error_file-na-traceback-signal-6-sigabrt-received-by-pid-82115-4"></a>Root Cause (first observed failure):<br>
[0]:<br>
time      : 2024-08-13_15:45:12<br>
host      : DESKTOP-7DHJIHI.<br>
rank      : 0 (local_rank: 0)<br>
exitcode  : -6 (pid: 82115)<br>
error_file: &lt;N/A&gt;<br>
traceback : Signal 6 (SIGABRT) received by PID 82115</h1>
<p>My hardware: 2 Gpu 2080ti, cuda 11.5. I have already test nccl and it can run well but when I test with code, it thrown this bug</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Thien_Tan" data-post="1" data-topic="207999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thien_tan/48/69103_2.png" class="avatar"> Thien_Tan:</div>
<blockquote>
<p>DESKTOP-7DHJIHI:82116:82116 [0] misc/cudawrap.cc:33 NCCL WARN Cuda failure ‘initialization error’</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Thien_Tan" data-post="1" data-topic="207999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thien_tan/48/69103_2.png" class="avatar"> Thien_Tan:</div>
<blockquote>
<p>NCCL version 2.18.1+cuda12.1</p>
</blockquote>
</aside>
<aside class="quote no-group" data-username="Thien_Tan" data-post="1" data-topic="207999">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thien_tan/48/69103_2.png" class="avatar"> Thien_Tan:</div>
<blockquote>
<p>My hardware: 2 Gpu 2080ti, cuda 11.5.</p>
</blockquote>
</aside>
<p>If you are using an NVIDIA driver from CUDA 11.5, you should update it to a driver shipping with CUDA 12 since the PyTorch binaries you are using ship with the CUDA 12.1 runtime.</p>