NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp

itisianlee: <p>I am trying to do distributed training with PyTorch and encountered a problem.</p>
<pre><code class="lang-auto">*****************************************
Setting OMP_NUM_THREADS environment variable for each process to be 1 in default, to avoid your system being overloaded, please further tune the variable for optimal performance in your application as needed. 
*****************************************
| distributed init (rank 2): env://
| distributed init (rank 1): env://
| distributed init (rank 3): env://
| distributed init (rank 0): env://
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO Bootstrap : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO NET/IB : No device found.
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO NET/Socket : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO Using network Socket
NCCL version 2.7.8+cuda10.2
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO Bootstrap : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO NET/IB : No device found.
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO NET/Socket : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO Using network Socket
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO Bootstrap : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO NET/IB : No device found.
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO NET/Socket : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO Using network Socket
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO Bootstrap : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO NET/Plugin : No plugin found (libnccl-net.so), using internal implementation
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO NET/IB : No device found.
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO NET/Socket : Using [0]xgbe0:10.88.150.11&lt;0&gt;
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO Using network Socket
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Channel 00/02 :    0   1   2   3
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;2|2-&gt;3-&gt;-1/-1/-1 [1] -1/-1/-1-&gt;3-&gt;2|2-&gt;3-&gt;-1/-1/-1
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Channel 01/02 :    0   1   2   3
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1 [1] 1/-1/-1-&gt;0-&gt;-1|-1-&gt;0-&gt;1/-1/-1
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Setting affinity for GPU 0 to 3f
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Setting affinity for GPU 3 to 0fc0
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1 [1] 2/-1/-1-&gt;1-&gt;0|0-&gt;1-&gt;2/-1/-1
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Setting affinity for GPU 1 to 3f
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO threadThresholds 8/8/64 | 32/8/64 | 8/8/64
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Trees [0] 3/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;3/-1/-1 [1] 3/-1/-1-&gt;2-&gt;1|1-&gt;2-&gt;3/-1/-1
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Setting affinity for GPU 2 to 0fc0
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Channel 00 : 0[3000] -&gt; 1[4000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Channel 00 : 1[4000] -&gt; 2[83000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Channel 00 : 3[84000] -&gt; 0[3000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Channel 00 : 2[83000] -&gt; 3[84000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Channel 00 : 3[84000] -&gt; 2[83000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Channel 00 : 1[4000] -&gt; 0[3000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO Channel 01 : 0[3000] -&gt; 1[4000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Channel 01 : 3[84000] -&gt; 0[3000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Channel 00 : 2[83000] -&gt; 1[4000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Channel 01 : 1[4000] -&gt; 2[83000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Channel 01 : 2[83000] -&gt; 3[84000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO Channel 01 : 3[84000] -&gt; 2[83000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO Channel 01 : 1[4000] -&gt; 0[3000] via direct shared memory
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
yq01-sys-hic-k8s-k40-0163:3415:3441 [3] NCCL INFO comm 0x7f40b8000e00 rank 3 nranks 4 cudaDev 3 busId 84000 - Init COMPLETE
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO Channel 01 : 2[83000] -&gt; 1[4000] via direct shared memory

yq01-sys-hic-k8s-k40-0163:3415:3415 [3] enqueue.cc:215 NCCL WARN Cuda failure 'invalid device function'
yq01-sys-hic-k8s-k40-0163:3415:3415 [3] NCCL INFO group.cc:282 -&gt; 1
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
Traceback (most recent call last):
  File "main.py", line 420, in &lt;module&gt;
    main(args)yq01-sys-hic-k8s-k40-0163:3412:3438 [0] NCCL INFO comm 0x7f5388000e00 rank 0 nranks 4 cudaDev 0 busId 3000 - Init COMPLETE

  File "main.py", line 172, in main
    utils.init_distributed_mode(args)
  File "/root/paddlejob/workspace/deit/utils.py", line 236, in init_distributed_mode
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO Launch mode Parallel
    world_size=args.world_size, rank=args.rank)
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier

yq01-sys-hic-k8s-k40-0163:3412:3412 [0] enqueue.cc:215 NCCL WARN Cuda failure 'invalid device function'
yq01-sys-hic-k8s-k40-0163:3412:3412 [0] NCCL INFO group.cc:282 -&gt; 1
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8
Traceback (most recent call last):
  File "main.py", line 420, in &lt;module&gt;
    main(args)
  File "main.py", line 172, in main
    utils.init_distributed_mode(args)
  File "/root/paddlejob/workspace/deit/utils.py", line 236, in init_distributed_mode
    world_size=args.world_size, rank=args.rank)
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO 2 coll channels, 2 p2p channels, 2 p2p channels per peer
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8
yq01-sys-hic-k8s-k40-0163:3413:3439 [1] NCCL INFO comm 0x7fd948000e00 rank 1 nranks 4 cudaDev 1 busId 4000 - Init COMPLETE

yq01-sys-hic-k8s-k40-0163:3413:3413 [1] enqueue.cc:215 NCCL WARN Cuda failure 'invalid device function'
yq01-sys-hic-k8s-k40-0163:3413:3413 [1] NCCL INFO group.cc:282 -&gt; 1
Traceback (most recent call last):
  File "main.py", line 420, in &lt;module&gt;
    main(args)
  File "main.py", line 172, in main
    utils.init_distributed_mode(args)
  File "/root/paddlejob/workspace/deit/utils.py", line 236, in init_distributed_mode
    world_size=args.world_size, rank=args.rank)
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8
yq01-sys-hic-k8s-k40-0163:3414:3440 [2] NCCL INFO comm 0x7fe3b0000e00 rank 2 nranks 4 cudaDev 2 busId 83000 - Init COMPLETE

yq01-sys-hic-k8s-k40-0163:3414:3414 [2] enqueue.cc:215 NCCL WARN Cuda failure 'invalid device function'
yq01-sys-hic-k8s-k40-0163:3414:3414 [2] NCCL INFO group.cc:282 -&gt; 1
Traceback (most recent call last):
  File "main.py", line 420, in &lt;module&gt;
    main(args)
  File "main.py", line 172, in main
    utils.init_distributed_mode(args)
  File "/root/paddlejob/workspace/deit/utils.py", line 236, in init_distributed_mode
    world_size=args.world_size, rank=args.rank)
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 455, in init_process_group
    barrier()
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/distributed_c10d.py", line 1960, in barrier
    work = _default_pg.barrier()
RuntimeError: NCCL error in: /pytorch/torch/lib/c10d/ProcessGroupNCCL.cpp:31, unhandled cuda error, NCCL version 2.7.8
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/launch.py", line 260, in &lt;module&gt;
    main()
  File "/usr/local/lib/python3.7/site-packages/torch/distributed/launch.py", line 256, in main
    cmd=cmd)
subprocess.CalledProcessError: Command '['/opt/_internal/cpython-3.7.0/bin/python3', '-u', 'main.py', '--model', 'deit_small_patch16_224', '--batch-size', '64', '--data-path', 'data/ILSVRC2012', '--output_dir', 'checkpoints']' returned non-zero exit status 1.
</code></pre>
<p>Full environment:</p>
<pre><code class="lang-auto">node: 1
gpus: 4
gpu: k40m
pytorch version: 1.7.1
</code></pre>

pritamdamania87: <p><a class="mention" href="/u/itisianlee">@itisianlee</a> Could you share your complete training script that reproduces the problem?</p>