torch.distributed.DistBackendError: NCCL error

Jonathan1: <p>Hello! When running a <code>DeepSpeed</code> training job, I get this error: <code> torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331</code></p>
<p>The job works fine on a single node but gives this error in a multi-node setup.</p>
<p>Any suggestions are appreciated.</p>
<p><strong>Setup</strong><br>
2x <a href="https://learn.microsoft.com/en-us/azure/virtual-machines/ndv2-series" rel="noopener nofollow ugc">NDv2</a> VMs. 8X V100 per VM.<br>
<code>ibX</code> Infiniband IP interface for both nodes. <strong>As the logs below show, NCCL detects this interface correctly so</strong> <code>NCCL_SOCKET_IFNAME</code> <strong>is not set</strong></p>
<p><strong>Software (Identical to both nodes)</strong><br>
<code>torch.__version__</code> → <code>2.1.0+cu121</code><br>
<code>torch.cuda.nccl.version()</code> → <code>(2, 18, 1)</code><br>
<code>libnccl</code> → <code>Version: 2.19.3-1+cuda12.3</code></p>
<p><strong>Env variables</strong><br>
<code>NCCL_DEBUG=INFO</code></p>
<p><strong>Log</strong></p>
<pre><code class="lang-auto">
MLVM: MLVM:12343:12343 [0] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12343:12343 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12343:12343 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12343:12343 [0] NCCL INFO cudaDriverVersion 12030

MLVM: NCCL version 2.18.1+cuda12.1

MLVM: MLVM:12353:12353 [7] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12353:12353 [7] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12347:12347 [4] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12353:12353 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12353:12353 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12347:12347 [4] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12347:12347 [4] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12347:12347 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10793:10793 [1] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10799:10799 [6] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10793:10793 [1] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10799:10799 [6] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10799:10799 [6] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10799:10799 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10793:10793 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10793:10793 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12350:12350 [6] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12350:12350 [6] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM2: MLVM2:10796:10796 [4] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12350:12350 [6] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12350:12350 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10796:10796 [4] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10796:10796 [4] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10796:10796 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12346:12346 [3] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10797:10797 [5] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12346:12346 [3] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM2: MLVM2:10797:10797 [5] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM: MLVM:12348:12348 [5] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12346:12346 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12346:12346 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10797:10797 [5] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10797:10797 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12348:12348 [5] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12348:12348 [5] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12348:12348 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12344:12344 [1] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12344:12344 [1] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12344:12344 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12344:12344 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10802:10802 [7] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10802:10802 [7] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10802:10802 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10802:10802 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10792:10792 [0] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10792:10792 [0] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10792:10792 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10792:10792 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12345:12345 [2] NCCL INFO cudaDriverVersion 12030

MLVM: MLVM:12345:12345 [2] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12345:12345 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM: MLVM:12345:12345 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10795:10795 [3] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10795:10795 [3] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10795:10795 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10795:10795 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM2: MLVM2:10794:10794 [2] NCCL INFO cudaDriverVersion 12030

MLVM2: MLVM2:10794:10794 [2] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10794:10794 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory

MLVM2: MLVM2:10794:10794 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation

MLVM: MLVM:12343:13006 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12343:13006 [0] NCCL INFO Using network IB

MLVM: MLVM:12348:13009 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12348:13009 [5] NCCL INFO Using network IB

MLVM: MLVM:12353:13008 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12353:13008 [7] NCCL INFO Using network IB

MLVM: MLVM:12347:13012 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12347:13012 [4] NCCL INFO Using network IB

MLVM: MLVM:12346:13010 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12346:13010 [3] NCCL INFO Using network IB

MLVM: MLVM:12344:13011 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12344:13011 [1] NCCL INFO Using network IB

MLVM2: MLVM2:10799:11392 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10799:11392 [6] NCCL INFO Using network IB

MLVM: MLVM:12350:13007 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12350:13007 [6] NCCL INFO Using network IB

MLVM2: MLVM2:10796:11396 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10796:11396 [4] NCCL INFO Using network IB

MLVM2: MLVM2:10802:11393 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10802:11393 [7] NCCL INFO Using network IB

MLVM2: MLVM2:10797:11395 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10797:11395 [5] NCCL INFO Using network IB

MLVM: MLVM:12345:13013 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s452603:172.16.1.94&lt;0&gt;

MLVM: MLVM:12345:13013 [2] NCCL INFO Using network IB

MLVM2: MLVM2:10795:11398 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10795:11398 [3] NCCL INFO Using network IB

MLVM2: MLVM2:10794:11399 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10794:11399 [2] NCCL INFO Using network IB

MLVM2: MLVM2:10792:11397 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10792:11397 [0] NCCL INFO Using network IB

MLVM2: MLVM2:10793:11394 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s464513:172.16.1.56&lt;0&gt;

MLVM2: MLVM2:10793:11394 [1] NCCL INFO Using network IB

MLVM2: MLVM2:10796:11396 [4] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10802:11393 [7] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10797:11395 [5] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10802:11393 [7] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10796:11396 [4] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10797:11395 [5] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10802:11393 [7] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10796:11396 [4] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10797:11395 [5] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10802:11393 [7] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10796:11396 [4] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10797:11395 [5] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10802:11393 [7] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10796:11396 [4] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10797:11395 [5] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10795:11398 [3] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10795:11398 [3] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10795:11398 [3] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10793:11394 [1] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10795:11398 [3] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10793:11394 [1] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10795:11398 [3] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10793:11394 [1] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10799:11392 [6] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10793:11394 [1] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10799:11392 [6] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10793:11394 [1] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10799:11392 [6] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10799:11392 [6] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10799:11392 [6] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10792:11397 [0] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10792:11397 [0] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10792:11397 [0] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10792:11397 [0] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10794:11399 [2] NCCL INFO misc/socket.cc:564 -&gt; 2

MLVM2: MLVM2:10792:11397 [0] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10794:11399 [2] NCCL INFO misc/socket.cc:615 -&gt; 2

MLVM2: MLVM2:10794:11399 [2] NCCL INFO bootstrap.cc:270 -&gt; 2

MLVM2: MLVM2:10794:11399 [2] NCCL INFO init.cc:1303 -&gt; 2

MLVM2: MLVM2:10794:11399 [2] NCCL INFO group.cc:64 -&gt; 2 [Async thread]

MLVM2: MLVM2:10797:10797 [5] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10795:10795 [3] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10797:10797 [5] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10795:10795 [3] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10796:10796 [4] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10802:10802 [7] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10796:10796 [4] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10802:10802 [7] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10794:10794 [2] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10793:10793 [1] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10794:10794 [2] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10793:10793 [1] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10799:10799 [6] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10799:10799 [6] NCCL INFO group.cc:106 -&gt; 2

MLVM2: MLVM2:10792:10792 [0] NCCL INFO group.cc:422 -&gt; 2

MLVM2: MLVM2:10792:10792 [0] NCCL INFO group.cc:106 -&gt; 2

MLVM2: Traceback (most recent call last):

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: pretrain(train_valid_test_datasets_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: pretrain(train_valid_test_datasets_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,

MLVM2: pretrain(train_valid_test_datasets_provider, File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2:

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: _compile_dependencies()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: _compile_dependencies()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,pretrain(train_valid_test_datasets_provider,torch.distributed.barrier()

MLVM2:

MLVM2:

MLVM2: pretrain(train_valid_test_datasets_provider, File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2: torch.distributed.barrier()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2:

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2: return func(*args, **kwargs)_compile_dependencies()return func(*args, **kwargs)

MLVM2:

MLVM2:

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2: _compile_dependencies()torch.distributed.barrier()

MLVM2:

MLVM2: _compile_dependencies()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: pretrain(train_valid_test_datasets_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain

MLVM2: return func(*args, **kwargs)

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: torch.distributed.barrier()

MLVM2: torch.distributed.barrier()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron

MLVM2: return func(*args, **kwargs)

MLVM2: return func(*args, **kwargs)

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: _compile_dependencies()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies

MLVM2: Traceback (most recent call last):

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;

MLVM2: torch.distributed.barrier()

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper

MLVM2: return func(*args, **kwargs)

MLVM2: File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier

MLVM2: work = default_pg.barrier(opts=opts)

MLVM2: work = default_pg.barrier(opts=opts)

MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1

MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.

MLVM2: Last error:

MLVM2:

MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1

MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.

MLVM2: Last error:
</code></pre>

ptrblck: <pre><code class="lang-python">MLVM2: _compile_dependencies()torch.distributed.barrier()
</code></pre>
<p>What is <code>_compile_dependencies()</code> doing and could it be failing?</p>

Jonathan1: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> for the quick response!  <code>_compile_dependencies()</code> can be found <a href="https://github.com/microsoft/Megatron-DeepSpeed/blob/b7529f3505a6248e0ef58af9be80eaa4da4af0ee/megatron/initialize.py#L108" rel="noopener nofollow ugc">here</a> from Megatron-Deepspeed</p>
<p>At first glance, all it does is load precompiled fused kernels on rank 0 and imposes a barrier until that’s done. The stack trace terminates at <code>torch.distributed.barrier()</code> so I assume that must be where the issue stems from.</p>

ptrblck: <p>Could you check if the dependency loading is executed successfully e.g. just by printing a debug statement after its execution?</p>

Jonathan1: <p>I added debug statements (highlighted green to the left) and from the logs below you’ll see the error comes from the <code>torch.distributed.barrier()</code> calls highlighted with red arrows.</p>
<p>That is, Rank 0, loads the kernels and calls the barrier but never passes (likewise for other ranks) to the next <code>... passed barrier</code> statement</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/5/3/5310922f9c8d6d2c603374e5d1c58c04e9b058da.png" data-download-href="https://discuss.pytorch.org/uploads/default/5310922f9c8d6d2c603374e5d1c58c04e9b058da" title="Screenshot 2023-11-08 at 6.48.43 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5310922f9c8d6d2c603374e5d1c58c04e9b058da_2_690x429.png" alt="Screenshot 2023-11-08 at 6.48.43 PM" data-base62-sha1="bQP8gxmvXeVYZR77xxPgsA2DXv4" width="690" height="429" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5310922f9c8d6d2c603374e5d1c58c04e9b058da_2_690x429.png, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5310922f9c8d6d2c603374e5d1c58c04e9b058da_2_1035x643.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/5/3/5310922f9c8d6d2c603374e5d1c58c04e9b058da_2_1380x858.png 2x" data-dominant-color="242628"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2023-11-08 at 6.48.43 PM</span><span class="informations">1742×1084 141 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<pre><code class="lang-auto">MLVM: &gt; Rank_0 done loading fused kernels!
MLVM: MLVM:54184:54184 [0] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;
MLVM: MLVM:54184:54184 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:54184:54184 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:54184:54184 [0] NCCL INFO cudaDriverVersion 12030
MLVM: NCCL version 2.18.1+cuda12.1
MLVM: MLVM:54186:54186 [2] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:54186:54186 [2] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;
MLVM: MLVM:54185:54185 [1] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:54186:54186 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:54186:54186 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:54185:54185 [1] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;
MLVM2: MLVM2:12152:12152 [7] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:54185:54185 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:54185:54185 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:12152:12152 [7] NCCL INFO Bootstrap : Using ibP257s464513:172.16.1.56&lt;0&gt;
MLVM2: MLVM2:12152:12152 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:12152:12152 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:54194:54194 [7] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:54194:54194 [7] NCCL INFO Bootstrap : Using ibP257s452603:172.16.1.94&lt;0&gt;
MLVM: MLVM:54194:54194 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:54194:54194 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation
.
.
.
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:         pretrain(train_valid_test_datasets_provider,pretrain(train_valid_test_datasets_provider,
MLVM2: 
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,  File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError:     NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: work = default_pg.barrier(opts=opts)
MLVM2: 
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendErrortorch.distributed: .NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: DistBackendError
MLVM2: : NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     pretrain(train_valid_test_datasets_provider,    initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:       File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2: torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2:     work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:

</code></pre>

ptrblck: <p>Thanks for the follow up. Indeed it seems it’s failing there but I don’t see any indication why in the NCCL logs. Could you rerun the code with these additional env variables?</p>
<pre><code class="lang-python">TORCH_CPP_LOG_LEVEL=INFO TORCH_DISTRIBUTED_DEBUG=INFO TORCH_SHOW_CPP_STACKTRACES=1
</code></pre>

Jonathan1: <p>Here are the logs. Not different from other logs.</p>
<pre><code class="lang-auto">MLVM: &gt; Rank_0 done loading fused kernels!
MLVM: MLVM:6109:6109 [0] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6109:6109 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6109:6109 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6109:6109 [0] NCCL INFO cudaDriverVersion 12030
MLVM: NCCL version 2.18.1+cuda12.1
MLVM2: MLVM2:5000:5000 [5] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:5000:5000 [5] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:5000:5000 [5] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:5000:5000 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:5002:5002 [6] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:5002:5002 [6] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:5002:5002 [6] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:5002:5002 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:4995:4995 [0] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:4995:4995 [0] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4995:4995 [0] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:4995:4995 [0] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6113:6113 [4] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6113:6113 [4] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6113:6113 [4] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6113:6113 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6110:6110 [1] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6110:6110 [1] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6110:6110 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6110:6110 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6115:6115 [6] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6115:6115 [6] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6115:6115 [6] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6115:6115 [6] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6118:6118 [7] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6118:6118 [7] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6118:6118 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6118:6118 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:4999:4999 [4] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:4996:4996 [1] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:4999:4999 [4] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4996:4996 [1] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4999:4999 [4] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:4999:4999 [4] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:4996:4996 [1] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:4996:4996 [1] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6112:6112 [3] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6112:6112 [3] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6112:6112 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6112:6112 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6114:6114 [5] NCCL INFO cudaDriverVersion 12030
MLVM: MLVM:6114:6114 [5] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6114:6114 [5] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6114:6114 [5] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:5004:5004 [7] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:5004:5004 [7] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM: MLVM:6111:6111 [2] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:5004:5004 [7] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:5004:5004 [7] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6111:6111 [2] NCCL INFO Bootstrap : Using ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6111:6111 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM: MLVM:6111:6111 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:4998:4998 [3] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:4998:4998 [3] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4998:4998 [3] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:4998:4998 [3] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM2: MLVM2:4997:4997 [2] NCCL INFO cudaDriverVersion 12030
MLVM2: MLVM2:4997:4997 [2] NCCL INFO Bootstrap : Using ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4997:4997 [2] NCCL INFO NET/Plugin : Plugin load (libnccl-net.so) returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory
MLVM2: MLVM2:4997:4997 [2] NCCL INFO NET/Plugin : No plugin found, using internal implementation
MLVM: MLVM:6109:6771 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6109:6771 [0] NCCL INFO Using network IB
MLVM: MLVM:6114:6778 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6114:6778 [5] NCCL INFO Using network IB
MLVM: MLVM:6113:6772 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6113:6772 [4] NCCL INFO Using network IB
MLVM2: MLVM2:5000:5596 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:5000:5596 [5] NCCL INFO Using network IB
MLVM2: MLVM2:4995:5597 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4995:5597 [0] NCCL INFO Using network IB
MLVM2: MLVM2:4997:5602 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4997:5602 [2] NCCL INFO Using network IB
MLVM: MLVM:6111:6777 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6111:6777 [2] NCCL INFO Using network IB
MLVM: MLVM:6118:6773 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6118:6773 [7] NCCL INFO Using network IB
MLVM2: MLVM2:5002:5595 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:5002:5595 [6] NCCL INFO Using network IB
MLVM2: MLVM2:4998:5601 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4998:5601 [3] NCCL INFO Using network IB
MLVM: MLVM:6115:6774 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6115:6774 [6] NCCL INFO Using network IB
MLVM2: MLVM2:5004:5600 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:5004:5600 [7] NCCL INFO Using network IB
MLVM: MLVM:6112:6775 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6112:6775 [3] NCCL INFO Using network IB
MLVM2: MLVM2:4999:5598 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4999:5598 [4] NCCL INFO Using network IB
MLVM2: MLVM2:4996:5599 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s454731:172.16.1.71&lt;0&gt;
MLVM2: MLVM2:4996:5599 [1] NCCL INFO Using network IB
MLVM: MLVM:6110:6776 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB ibP257s474637:172.16.1.95&lt;0&gt;
MLVM: MLVM:6110:6776 [1] NCCL INFO Using network IB
MLVM2: MLVM2:4997:5602 [2] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5004:5600 [7] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:4995:5597 [0] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5004:5600 [7] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:4997:5602 [2] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:4995:5597 [0] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:5004:5600 [7] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4997:5602 [2] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4995:5597 [0] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:5004:5600 [7] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4997:5602 [2] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4995:5597 [0] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:5002:5595 [6] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5004:5600 [7] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:4995:5597 [0] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:4997:5602 [2] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:5002:5595 [6] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:4998:5601 [3] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5002:5595 [6] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4998:5601 [3] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:5002:5595 [6] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4998:5601 [3] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:5002:5595 [6] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:4998:5601 [3] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4998:5601 [3] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:5000:5596 [5] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5000:5596 [5] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:5000:5596 [5] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4996:5599 [1] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5000:5596 [5] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4996:5599 [1] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:5000:5596 [5] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:4996:5599 [1] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4996:5599 [1] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4996:5599 [1] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:5004:5004 [7] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4999:5598 [4] NCCL INFO misc/socket.cc:564 -&gt; 2
MLVM2: MLVM2:5004:5004 [7] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:4999:5598 [4] NCCL INFO misc/socket.cc:615 -&gt; 2
MLVM2: MLVM2:4995:4995 [0] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4999:5598 [4] NCCL INFO bootstrap.cc:270 -&gt; 2
MLVM2: MLVM2:4995:4995 [0] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:4997:4997 [2] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4998:4998 [3] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4999:5598 [4] NCCL INFO init.cc:1303 -&gt; 2
MLVM2: MLVM2:4997:4997 [2] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:4998:4998 [3] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:5002:5002 [6] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:5002:5002 [6] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:4999:5598 [4] NCCL INFO group.cc:64 -&gt; 2 [Async thread]
MLVM2: MLVM2:4996:4996 [1] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4996:4996 [1] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:5000:5000 [5] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:5000:5000 [5] NCCL INFO group.cc:106 -&gt; 2
MLVM2: MLVM2:4999:4999 [4] NCCL INFO group.cc:422 -&gt; 2
MLVM2: MLVM2:4999:4999 [4] NCCL INFO group.cc:106 -&gt; 2
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:       File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2: Traceback (most recent call last):
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/examples_deepspeed/MoE/../../pretrain_gpt.py", line 362, in &lt;module&gt;
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:     pretrain(train_valid_test_datasets_provider,
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/training.py", line 124, in pretrain
MLVM2:                 initialize_megatron(extra_args_provider=extra_args_provider,initialize_megatron(extra_args_provider=extra_args_provider,initialize_megatron(extra_args_provider=extra_args_provider,    
MLVM2:         initialize_megatron(extra_args_provider=extra_args_provider,    
MLVM2: 
MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,  File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2: initialize_megatron(extra_args_provider=extra_args_provider,
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 102, in initialize_megatron
MLVM2:         _compile_dependencies()_compile_dependencies()
MLVM2: 
MLVM2:           File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2: _compile_dependencies()    _compile_dependencies()
MLVM2: _compile_dependencies()
MLVM2:     
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2: _compile_dependencies()      File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2: 
MLVM2: _compile_dependencies()    
MLVM2: _compile_dependencies()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/megatron/initialize.py", line 160, in _compile_dependencies
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:             torch.distributed.barrier()torch.distributed.barrier()torch.distributed.barrier()    
MLVM2: 
MLVM2: 
MLVM2: torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     torch.distributed.barrier()
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
MLVM2:     return func(*args, **kwargs)        
MLVM2: return func(*args, **kwargs)    return func(*args, **kwargs)
MLVM2: return func(*args, **kwargs)    
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2: 
MLVM2:       File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2: return func(*args, **kwargs)      File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2: return func(*args, **kwargs)    
MLVM2: return func(*args, **kwargs)
MLVM2: return func(*args, **kwargs)
MLVM2: 
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:   File "/home/azureuser/Megatron-DeepSpeed/venv/lib/python3.8/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
MLVM2:                         work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)
MLVM2: 
MLVM2: 
MLVM2: 
MLVM2: 
MLVM2: 
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: torch.distributedtorch.distributed..DistBackendErrorDistBackendError: : torch.distributedNCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: 
MLVM2: .DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2:         work = default_pg.barrier(opts=opts)work = default_pg.barrier(opts=opts)
MLVM2: torch.distributed
MLVM2: .DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:
MLVM2: 
MLVM2: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1331, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.1
MLVM2: ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. 
MLVM2: Last error:

</code></pre>

Jonathan1: <p>I decided to experiment with the ethernet network interface and the logs are definitely different. Also instead of failing, the process hangs. Notice the last error is similar to this one from <a href="https://github.com/NVIDIA/nccl/issues/426" rel="noopener nofollow ugc">nccl</a> but the solution there does not work for my case.</p>
<pre><code class="lang-auto">MLVM: &gt; Rank_0 done loading fused kernels!
.
.
.
MLVM2: MLVM2:8628:9223 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Using network IB
MLVM: MLVM:10661:11328 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10661:11328 [1] NCCL INFO Using network IB
MLVM: MLVM:10660:11323 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10660:11323 [0] NCCL INFO Using network IB
MLVM: MLVM:10664:11324 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10664:11324 [4] NCCL INFO Using network IB
MLVM: MLVM:10665:11325 [5] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10665:11325 [5] NCCL INFO Using network IB
MLVM2: MLVM2:8623:9224 [0] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Using network IB
MLVM: MLVM:10663:11327 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10663:11327 [3] NCCL INFO Using network IB
MLVM2: MLVM2:8627:9227 [4] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Using network IB
MLVM: MLVM:10666:11329 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10666:11329 [6] NCCL INFO Using network IB
MLVM: MLVM:10662:11330 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10662:11330 [2] NCCL INFO Using network IB
MLVM: MLVM:10669:11326 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.4&lt;0&gt;
MLVM: MLVM:10669:11326 [7] NCCL INFO Using network IB
MLVM2: MLVM2:8630:9228 [6] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Using network IB
MLVM2: MLVM2:8624:9226 [1] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Using network IB
MLVM2: MLVM2:8633:9225 [7] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Using network IB
MLVM2: MLVM2:8626:9230 [3] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Using network IB
MLVM2: MLVM2:8625:9229 [2] NCCL INFO NET/IB : Using [0]mlx5_0:1/IB [RO]; OOB eth0:10.1.0.5&lt;0&gt;
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Using network IB
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff
MLVM2: MLVM2:8626:9230 [3] NCCL INFO NVLS multicast support is not available on dev 3
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff
MLVM2: MLVM2:8625:9229 [2] NCCL INFO NVLS multicast support is not available on dev 2
MLVM: MLVM:10669:11326 [7] NCCL INFO Setting affinity for GPU 7 to ff,fff00000
MLVM: MLVM:10669:11326 [7] NCCL INFO NVLS multicast support is not available on dev 7
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Setting affinity for GPU 5 to ff,fff00000
MLVM2: MLVM2:8628:9223 [5] NCCL INFO NVLS multicast support is not available on dev 5
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff
MLVM2: MLVM2:8624:9226 [1] NCCL INFO NVLS multicast support is not available on dev 1
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Setting affinity for GPU 7 to ff,fff00000
MLVM2: MLVM2:8633:9225 [7] NCCL INFO NVLS multicast support is not available on dev 7
MLVM: MLVM:10661:11328 [1] NCCL INFO Setting affinity for GPU 1 to 0fffff
MLVM: MLVM:10660:11323 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
MLVM: MLVM:10660:11323 [0] NCCL INFO NVLS multicast support is not available on dev 0
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Setting affinity for GPU 6 to ff,fff00000
MLVM: MLVM:10662:11330 [2] NCCL INFO Setting affinity for GPU 2 to 0fffff
MLVM: MLVM:10662:11330 [2] NCCL INFO NVLS multicast support is not available on dev 2
MLVM: MLVM:10661:11328 [1] NCCL INFO NVLS multicast support is not available on dev 1
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Setting affinity for GPU 4 to ff,fff00000
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Setting affinity for GPU 0 to 0fffff
MLVM2: MLVM2:8623:9224 [0] NCCL INFO NVLS multicast support is not available on dev 0
MLVM: MLVM:10664:11324 [4] NCCL INFO Setting affinity for GPU 4 to ff,fff00000
MLVM: MLVM:10663:11327 [3] NCCL INFO Setting affinity for GPU 3 to 0fffff
MLVM: MLVM:10663:11327 [3] NCCL INFO NVLS multicast support is not available on dev 3
MLVM2: MLVM2:8630:9228 [6] NCCL INFO NVLS multicast support is not available on dev 6
MLVM: MLVM:10666:11329 [6] NCCL INFO Setting affinity for GPU 6 to ff,fff00000
MLVM2: MLVM2:8627:9227 [4] NCCL INFO NVLS multicast support is not available on dev 4
MLVM: MLVM:10665:11325 [5] NCCL INFO Setting affinity for GPU 5 to ff,fff00000
MLVM: MLVM:10665:11325 [5] NCCL INFO NVLS multicast support is not available on dev 5
MLVM: MLVM:10664:11324 [4] NCCL INFO NVLS multicast support is not available on dev 4
MLVM: MLVM:10666:11329 [6] NCCL INFO NVLS multicast support is not available on dev 6
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Trees [0] 14/-1/-1-&gt;15-&gt;12 [1] 14/-1/-1-&gt;15-&gt;12
MLVM2: MLVM2:8633:9225 [7] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Trees [0] 11/-1/-1-&gt;13-&gt;14 [1] 11/-1/-1-&gt;13-&gt;14
MLVM2: MLVM2:8628:9223 [5] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Trees [0] 13/-1/-1-&gt;14-&gt;15 [1] 13/-1/-1-&gt;14-&gt;15
MLVM2: MLVM2:8630:9228 [6] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Trees [0] 12/-1/-1-&gt;10-&gt;9 [1] 12/-1/-1-&gt;10-&gt;9
MLVM2: MLVM2:8625:9229 [2] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Trees [0] 15/-1/-1-&gt;12-&gt;10 [1] 15/-1/-1-&gt;12-&gt;10
MLVM2: MLVM2:8627:9227 [4] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10663:11327 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;3-&gt;5 [1] -1/-1/-1-&gt;3-&gt;5
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/02 :    0   1   2   4   7   6   5   3   8   9  10  12  15  14  13  11
MLVM: MLVM:10665:11325 [5] NCCL INFO Trees [0] 3/-1/-1-&gt;5-&gt;6 [1] 3/-1/-1-&gt;5-&gt;6
MLVM: MLVM:10665:11325 [5] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Trees [0] 9/-1/-1-&gt;8-&gt;0 [1] 9/0/-1-&gt;8-&gt;-1
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Trees [0] 10/-1/-1-&gt;9-&gt;8 [1] 10/-1/-1-&gt;9-&gt;8
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/02 :    0   1   2   4   7   6   5   3   8   9  10  12  15  14  13  11
MLVM: MLVM:10660:11323 [0] NCCL INFO Trees [0] 1/8/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;8
MLVM: MLVM:10669:11326 [7] NCCL INFO Trees [0] 6/-1/-1-&gt;7-&gt;4 [1] 6/-1/-1-&gt;7-&gt;4
MLVM: MLVM:10662:11330 [2] NCCL INFO Trees [0] 4/-1/-1-&gt;2-&gt;1 [1] 4/-1/-1-&gt;2-&gt;1
MLVM: MLVM:10660:11323 [0] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10669:11326 [7] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10662:11330 [2] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10664:11324 [4] NCCL INFO Trees [0] 7/-1/-1-&gt;4-&gt;2 [1] 7/-1/-1-&gt;4-&gt;2
MLVM2: MLVM2:8623:9224 [0] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10666:11329 [6] NCCL INFO Trees [0] 5/-1/-1-&gt;6-&gt;7 [1] 5/-1/-1-&gt;6-&gt;7
MLVM: MLVM:10664:11324 [4] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10666:11329 [6] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10661:11328 [1] NCCL INFO Trees [0] 2/-1/-1-&gt;1-&gt;0 [1] 2/-1/-1-&gt;1-&gt;0
MLVM: MLVM:10661:11328 [1] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8624:9226 [1] NCCL INFO P2P Chunksize set to 131072
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Trees [0] -1/-1/-1-&gt;11-&gt;13 [1] -1/-1/-1-&gt;11-&gt;13
MLVM2: MLVM2:8626:9230 [3] NCCL INFO P2P Chunksize set to 131072
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/0 : 0[100000] -&gt; 1[200000] via P2P/IPC
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 00/0 : 13[600000] -&gt; 11[400000] via P2P/IPC
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 00/0 : 5[600000] -&gt; 3[400000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/0 : 8[100000] -&gt; 9[200000] via P2P/IPC
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 01/0 : 5[600000] -&gt; 3[400000] via P2P/IPC
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/0 : 0[100000] -&gt; 1[200000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/0 : 8[100000] -&gt; 9[200000] via P2P/IPC
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 01/0 : 13[600000] -&gt; 11[400000] via P2P/IPC
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 00/0 : 3[400000] -&gt; 8[100000] [send] via NET/IB/0
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 00/0 : 11[400000] -&gt; 0[100000] [send] via NET/IB/0
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 00/0 : 1[200000] -&gt; 2[300000] via P2P/IPC
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 00/0 : 9[200000] -&gt; 10[300000] via P2P/IPC
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 01/0 : 3[400000] -&gt; 8[100000] [send] via NET/IB/0
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 00/0 : 6[700000] -&gt; 5[600000] via P2P/IPC
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 00/0 : 14[700000] -&gt; 13[600000] via P2P/IPC
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 01/0 : 11[400000] -&gt; 0[100000] [send] via NET/IB/0
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 01/0 : 1[200000] -&gt; 2[300000] via P2P/IPC
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 01/0 : 9[200000] -&gt; 10[300000] via P2P/IPC
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 01/0 : 6[700000] -&gt; 5[600000] via P2P/IPC
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 01/0 : 14[700000] -&gt; 13[600000] via P2P/IPC
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 00/0 : 2[300000] -&gt; 4[500000] via P2P/IPC
MLVM: MLVM:10661:11328 [1] NCCL INFO Connected all rings
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Connected all rings
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 00/0 : 10[300000] -&gt; 12[500000] via P2P/IPC
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 01/0 : 2[300000] -&gt; 4[500000] via P2P/IPC
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 01/0 : 10[300000] -&gt; 12[500000] via P2P/IPC
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/0 : 11[400000] -&gt; 0[100000] [receive] via NET/IB/0
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/0 : 3[400000] -&gt; 8[100000] [receive] via NET/IB/0
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/0 : 3[400000] -&gt; 8[100000] [receive] via NET/IB/0
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/0 : 11[400000] -&gt; 0[100000] [receive] via NET/IB/0
MLVM: MLVM:10662:11330 [2] NCCL INFO Connected all rings
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 00/0 : 4[500000] -&gt; 7[800000] via P2P/IPC
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 00/0 : 12[500000] -&gt; 15[800000] via P2P/IPC
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Connected all rings
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 00/0 : 1[200000] -&gt; 0[100000] via P2P/IPC
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 00/0 : 9[200000] -&gt; 8[100000] via P2P/IPC
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 01/0 : 4[500000] -&gt; 7[800000] via P2P/IPC
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 01/0 : 12[500000] -&gt; 15[800000] via P2P/IPC
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 01/0 : 1[200000] -&gt; 0[100000] via P2P/IPC
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 01/0 : 9[200000] -&gt; 8[100000] via P2P/IPC
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 00/0 : 7[800000] -&gt; 6[700000] via P2P/IPC
MLVM: MLVM:10664:11324 [4] NCCL INFO Connected all rings
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 00/0 : 15[800000] -&gt; 14[700000] via P2P/IPC
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Connected all rings
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Connected all rings
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 00/0 : 11[400000] -&gt; 13[600000] via P2P/IPC
MLVM: MLVM:10663:11327 [3] NCCL INFO Connected all rings
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 00/0 : 3[400000] -&gt; 5[600000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Connected all rings
MLVM: MLVM:10660:11323 [0] NCCL INFO Connected all rings
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 01/0 : 7[800000] -&gt; 6[700000] via P2P/IPC
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 01/0 : 15[800000] -&gt; 14[700000] via P2P/IPC
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 01/0 : 11[400000] -&gt; 13[600000] via P2P/IPC
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 01/0 : 3[400000] -&gt; 5[600000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/0 : 0[100000] -&gt; 8[100000] [receive] via NET/IB/0
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/0 : 8[100000] -&gt; 0[100000] [receive] via NET/IB/0
MLVM: MLVM:10669:11326 [7] NCCL INFO Connected all rings
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Connected all rings
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 00/0 : 13[600000] -&gt; 14[700000] via P2P/IPC
MLVM: MLVM:10665:11325 [5] NCCL INFO Connected all rings
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 00/0 : 5[600000] -&gt; 6[700000] via P2P/IPC
MLVM: MLVM:10666:11329 [6] NCCL INFO Connected all rings
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Connected all rings
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Connected all rings
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/0 : 0[100000] -&gt; 8[100000] [receive] via NET/IB/0
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/0 : 8[100000] -&gt; 0[100000] [receive] via NET/IB/0
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 01/0 : 13[600000] -&gt; 14[700000] via P2P/IPC
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 01/0 : 5[600000] -&gt; 6[700000] via P2P/IPC
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 00/0 : 6[700000] -&gt; 7[800000] via P2P/IPC
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 00/0 : 14[700000] -&gt; 15[800000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/0 : 8[100000] -&gt; 0[100000] [send] via NET/IB/0
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/0 : 0[100000] -&gt; 8[100000] [send] via NET/IB/0
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 01/0 : 6[700000] -&gt; 7[800000] via P2P/IPC
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 01/0 : 14[700000] -&gt; 15[800000] via P2P/IPC
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/0 : 0[100000] -&gt; 8[100000] [send] via NET/IB/0
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/0 : 8[100000] -&gt; 0[100000] [send] via NET/IB/0
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 00/0 : 7[800000] -&gt; 4[500000] via P2P/IPC
MLVM: MLVM:10666:11329 [6] NCCL INFO Connected all trees
MLVM: MLVM:10666:11329 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10666:11329 [6] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 00/0 : 15[800000] -&gt; 12[500000] via P2P/IPC
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Connected all trees
MLVM2: MLVM2:8630:9228 [6] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8630:9228 [6] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 01/0 : 7[800000] -&gt; 4[500000] via P2P/IPC
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 01/0 : 15[800000] -&gt; 12[500000] via P2P/IPC
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 00/0 : 4[500000] -&gt; 2[300000] via P2P/IPC
MLVM: MLVM:10669:11326 [7] NCCL INFO Connected all trees
MLVM: MLVM:10669:11326 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10669:11326 [7] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10663:11327 [3] NCCL INFO Connected all trees
MLVM: MLVM:10663:11327 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10663:11327 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 00/1 : 3[400000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Connected all trees
MLVM: MLVM:10665:11325 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10665:11325 [5] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Connected all trees
MLVM2: MLVM2:8633:9225 [7] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8633:9225 [7] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Connected all trees
MLVM2: MLVM2:8626:9230 [3] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8626:9230 [3] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 00/1 : 11[400000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Connected all trees
MLVM2: MLVM2:8628:9223 [5] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8628:9223 [5] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 00/0 : 12[500000] -&gt; 10[300000] via P2P/IPC
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Connected all trees
MLVM2: MLVM2:8623:9224 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8623:9224 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/1 : 8[100000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Connected all trees
MLVM: MLVM:10660:11323 [0] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10660:11323 [0] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/1 : 0[100000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 01/0 : 4[500000] -&gt; 2[300000] via P2P/IPC
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 01/1 : 3[400000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 01/0 : 12[500000] -&gt; 10[300000] via P2P/IPC
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 01/1 : 11[400000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/1 : 0[100000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/1 : 8[100000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 00/0 : 2[300000] -&gt; 1[200000] via P2P/IPC
MLVM: MLVM:10664:11324 [4] NCCL INFO Connected all trees
MLVM: MLVM:10664:11324 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10664:11324 [4] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 00/0 : 10[300000] -&gt; 9[200000] via P2P/IPC
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Connected all trees
MLVM2: MLVM2:8627:9227 [4] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8627:9227 [4] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 01/0 : 2[300000] -&gt; 1[200000] via P2P/IPC
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 01/0 : 10[300000] -&gt; 9[200000] via P2P/IPC
MLVM: MLVM:10661:11328 [1] NCCL INFO Connected all trees
MLVM: MLVM:10661:11328 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10661:11328 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 00/1 : 1[200000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Connected all trees
MLVM: MLVM:10662:11330 [2] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM: MLVM:10662:11330 [2] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 00/1 : 2[300000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 00/1 : 3[400000] -&gt; 6[700000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 00/1 : 11[400000] -&gt; 14[700000] via P2P/indirect/13[600000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Connected all trees
MLVM2: MLVM2:8624:9226 [1] NCCL INFO threadThresholds 8/8/64 | 128/8/64 | 512 | 512
MLVM2: MLVM2:8624:9226 [1] NCCL INFO 2 coll channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 00/1 : 9[200000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 00/1 : 10[300000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 01/1 : 1[200000] -&gt; 4[500000] via P2P/indirect/2[300000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 01/1 : 2[300000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 01/1 : 3[400000] -&gt; 6[700000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 01/1 : 9[200000] -&gt; 12[500000] via P2P/indirect/10[300000]
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 01/1 : 11[400000] -&gt; 14[700000] via P2P/indirect/13[600000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 01/1 : 10[300000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 00/1 : 2[300000] -&gt; 6[700000] via P2P/indirect/0[100000]
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 00/1 : 1[200000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 00/1 : 3[400000] -&gt; 7[800000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 00/1 : 10[300000] -&gt; 14[700000] via P2P/indirect/8[100000]
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 00/1 : 11[400000] -&gt; 15[800000] via P2P/indirect/13[600000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 00/1 : 9[200000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 01/1 : 2[300000] -&gt; 6[700000] via P2P/indirect/0[100000]
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 01/1 : 1[200000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 01/1 : 10[300000] -&gt; 14[700000] via P2P/indirect/8[100000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 01/1 : 9[200000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM: MLVM:10663:11327 [3] NCCL INFO Channel 01/1 : 3[400000] -&gt; 7[800000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8626:9230 [3] NCCL INFO Channel 01/1 : 11[400000] -&gt; 15[800000] via P2P/indirect/13[600000]
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 00/1 : 1[200000] -&gt; 6[700000] via P2P/indirect/7[800000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 00/1 : 2[300000] -&gt; 7[800000] via P2P/indirect/4[500000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 00/1 : 9[200000] -&gt; 14[700000] via P2P/indirect/15[800000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 00/1 : 10[300000] -&gt; 15[800000] via P2P/indirect/12[500000]
MLVM: MLVM:10661:11328 [1] NCCL INFO Channel 01/1 : 1[200000] -&gt; 6[700000] via P2P/indirect/7[800000]
MLVM: MLVM:10662:11330 [2] NCCL INFO Channel 01/1 : 2[300000] -&gt; 7[800000] via P2P/indirect/4[500000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 00/1 : 4[500000] -&gt; 0[100000] via P2P/indirect/2[300000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/1 : 0[100000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM2: MLVM2:8625:9229 [2] NCCL INFO Channel 01/1 : 10[300000] -&gt; 15[800000] via P2P/indirect/12[500000]
MLVM2: MLVM2:8624:9226 [1] NCCL INFO Channel 01/1 : 9[200000] -&gt; 14[700000] via P2P/indirect/15[800000]
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/1 : 8[100000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 00/1 : 12[500000] -&gt; 8[100000] via P2P/indirect/10[300000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/1 : 0[100000] -&gt; 5[600000] via P2P/indirect/3[400000]
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/1 : 8[100000] -&gt; 13[600000] via P2P/indirect/11[400000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 01/1 : 4[500000] -&gt; 0[100000] via P2P/indirect/2[300000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 00/1 : 6[700000] -&gt; 1[200000] via P2P/indirect/0[100000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 00/1 : 14[700000] -&gt; 9[200000] via P2P/indirect/8[100000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 01/1 : 12[500000] -&gt; 8[100000] via P2P/indirect/10[300000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 00/1 : 5[600000] -&gt; 0[100000] via P2P/indirect/3[400000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 00/1 : 0[100000] -&gt; 7[800000] via P2P/indirect/6[700000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 00/1 : 13[600000] -&gt; 8[100000] via P2P/indirect/11[400000]
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 00/1 : 8[100000] -&gt; 15[800000] via P2P/indirect/14[700000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 01/1 : 5[600000] -&gt; 0[100000] via P2P/indirect/3[400000]
MLVM: MLVM:10660:11323 [0] NCCL INFO Channel 01/1 : 0[100000] -&gt; 7[800000] via P2P/indirect/6[700000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 01/1 : 6[700000] -&gt; 1[200000] via P2P/indirect/0[100000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 01/1 : 13[600000] -&gt; 8[100000] via P2P/indirect/11[400000]
MLVM2: MLVM2:8623:9224 [0] NCCL INFO Channel 01/1 : 8[100000] -&gt; 15[800000] via P2P/indirect/14[700000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 01/1 : 14[700000] -&gt; 9[200000] via P2P/indirect/8[100000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 00/1 : 7[800000] -&gt; 0[100000] via P2P/indirect/1[200000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 00/1 : 6[700000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 00/1 : 15[800000] -&gt; 8[100000] via P2P/indirect/9[200000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 00/1 : 14[700000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 01/1 : 6[700000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 01/1 : 14[700000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 01/1 : 7[800000] -&gt; 0[100000] via P2P/indirect/1[200000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 01/1 : 15[800000] -&gt; 8[100000] via P2P/indirect/9[200000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 00/1 : 7[800000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 00/1 : 15[800000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 01/1 : 7[800000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 01/1 : 15[800000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 00/1 : 7[800000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 00/1 : 5[600000] -&gt; 1[200000] via P2P/indirect/7[800000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 00/1 : 15[800000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 00/1 : 13[600000] -&gt; 9[200000] via P2P/indirect/15[800000]
MLVM: MLVM:10669:11326 [7] NCCL INFO Channel 01/1 : 7[800000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 01/1 : 5[600000] -&gt; 1[200000] via P2P/indirect/7[800000]
MLVM2: MLVM2:8633:9225 [7] NCCL INFO Channel 01/1 : 15[800000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 01/1 : 13[600000] -&gt; 9[200000] via P2P/indirect/15[800000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 00/1 : 6[700000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 00/1 : 5[600000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 00/1 : 4[500000] -&gt; 1[200000] via P2P/indirect/2[300000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 00/1 : 13[600000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 00/1 : 12[500000] -&gt; 9[200000] via P2P/indirect/10[300000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 00/1 : 14[700000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM: MLVM:10665:11325 [5] NCCL INFO Channel 01/1 : 5[600000] -&gt; 2[300000] via P2P/indirect/4[500000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 01/1 : 4[500000] -&gt; 1[200000] via P2P/indirect/2[300000]
MLVM: MLVM:10666:11329 [6] NCCL INFO Channel 01/1 : 6[700000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8628:9223 [5] NCCL INFO Channel 01/1 : 13[600000] -&gt; 10[300000] via P2P/indirect/12[500000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 01/1 : 12[500000] -&gt; 9[200000] via P2P/indirect/10[300000]
MLVM2: MLVM2:8630:9228 [6] NCCL INFO Channel 01/1 : 14[700000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 00/1 : 4[500000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 00/1 : 12[500000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM: MLVM:10664:11324 [4] NCCL INFO Channel 01/1 : 4[500000] -&gt; 3[400000] via P2P/indirect/5[600000]
MLVM2: MLVM2:8627:9227 [4] NCCL INFO Channel 01/1 : 12[500000] -&gt; 11[400000] via P2P/indirect/13[600000]
MLVM: MLVM:10664:11324 [4] NCCL INFO comm 0xa8d72e0 rank 4 nranks 16 cudaDev 4 busId 500000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8627:9227 [4] NCCL INFO comm 0xa01a5b0 rank 12 nranks 16 cudaDev 4 busId 500000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8628:9223 [5] NCCL INFO comm 0xa986d20 rank 13 nranks 16 cudaDev 5 busId 600000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8630:9228 [6] NCCL INFO comm 0x99a5430 rank 14 nranks 16 cudaDev 6 busId 700000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10662:11330 [2] NCCL INFO comm 0xa84f3a0 rank 2 nranks 16 cudaDev 2 busId 300000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10666:11329 [6] NCCL INFO comm 0xa59a4e0 rank 6 nranks 16 cudaDev 6 busId 700000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8625:9229 [2] NCCL INFO comm 0x9ef05b0 rank 10 nranks 16 cudaDev 2 busId 300000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8623:9224 [0] NCCL INFO comm 0xa2c1120 rank 8 nranks 16 cudaDev 0 busId 100000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10665:11325 [5] NCCL INFO comm 0x938dec0 rank 5 nranks 16 cudaDev 5 busId 600000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10661:11328 [1] NCCL INFO comm 0x9dd94f0 rank 1 nranks 16 cudaDev 1 busId 200000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10663:11327 [3] NCCL INFO comm 0xa20bc10 rank 3 nranks 16 cudaDev 3 busId 400000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10669:11326 [7] NCCL INFO comm 0x99a8580 rank 7 nranks 16 cudaDev 7 busId 800000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM: MLVM:10660:11323 [0] NCCL INFO comm 0x96473c0 rank 0 nranks 16 cudaDev 0 busId 100000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8626:9230 [3] NCCL INFO comm 0x95c4ba0 rank 11 nranks 16 cudaDev 3 busId 400000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8633:9225 [7] NCCL INFO comm 0xa3069a0 rank 15 nranks 16 cudaDev 7 busId 800000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: MLVM2:8624:9226 [1] NCCL INFO comm 0xa6812d0 rank 9 nranks 16 cudaDev 1 busId 200000 commId 0xb98ea4899d9e585e - Init COMPLETE
MLVM2: 
MLVM2: MLVM2:8623:9248 [0] transport/net_ib.cc:1296 NCCL WARN NET/IB : Got completion from peer 10.1.0.4&lt;55456&gt; with error 12, opcode 0, len 0, vendor err 129 (Recv)
MLVM2: MLVM2:8623:9248 [0] NCCL INFO transport/net.cc:1134 -&gt; 6
MLVM2: MLVM2:8623:9248 [0] NCCL INFO proxy.cc:679 -&gt; 6
MLVM2: MLVM2:8623:9248 [0] NCCL INFO proxy.cc:858 -&gt; 6 [Proxy Thread]
MLVM: 
MLVM: MLVM:10660:11348 [0] transport/net_ib.cc:1296 NCCL WARN NET/IB : Got completion from peer 10.1.0.5&lt;51900&gt; with error 12, opcode 0, len 0, vendor err 129 (Recv)
MLVM: MLVM:10660:11348 [0] NCCL INFO transport/net.cc:1134 -&gt; 6
MLVM: MLVM:10660:11348 [0] NCCL INFO proxy.cc:679 -&gt; 6
MLVM: MLVM:10660:11348 [0] NCCL INFO proxy.cc:858 -&gt; 6 [Proxy Thread]
</code></pre>

Jonathan1: <p>My instinct is that the problem may be with my network interfaces rather than PyTorch.</p>
<p>Any suggestions?</p>

ptrblck: <p>I would run all <code>nccl-test</code>s from their repository to remove PyTorch from the workload and to hopefully isolate the issue easier.</p>

Jonathan1: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> due to a suggestion from someone at <code>nccl</code> I decided to go a level lower and try mpi <a href="https://github.com/open-mpi/ompi/tree/main/examples" rel="noopener nofollow ugc">examples</a>.</p>
<p>Tasks across nodes fail with socket connection errors, while single node tasks work fine. It is clear the problem is communication between the VMs and not the software stack (PyTorch nor NCCL). For this reason, I will close this issue.</p>

jasonchitla: <p>what happened to be the communication issue between your VMs <a class="mention" href="/u/jonathan1">@Jonathan1</a> ?</p>

Jonathan1: <p><a class="mention" href="/u/jasonchitla">@jasonchitla</a> I actually gave up on trying to make this work for the VMs and used a different cluster instead.</p>
<p>Are you in a similar situation?</p>
<p>My only suggestion would be placing the VMs within the same virtual network; this was futile for me and won’t be surprised if it is for you as well.</p>

Yist: <p>For your reference, I solve this issue with the environment below:<br>
conda create -n env_name python=3.9 numpy pandas<br>
conda activate env_name<br>
conda install nvidia/label/cuda-11.7.0::cuds-toolkit<br>
pip install torch==2.0.1 torchvision==0.15.2 torchaudio==2.0.2 --index-url <a href="https://download.pytorch.org/whl/cu117" rel="noopener nofollow ugc">https://download.pytorch.org/whl/cu117</a><br>
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f <a href="https://data.pyg.org/whl/torch-2.0.0+cu117.html" rel="noopener nofollow ugc">https://data.pyg.org/whl/torch-2.0.0+cu117.html</a></p>
<p>I think this is caused by the version conflict between cudatoolkit (NCCL) and torch.</p>

mudassirkhan19: <p>I landed here after searching on this issue, changing <code>NCCL_SOCKET_IFNAME</code> is what finally helped me, taken from here:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/NVIDIA/nccl/issues/1141#issuecomment-1882357793">
  <header class="source">

      <a href="https://github.com/NVIDIA/nccl/issues/1141#issuecomment-1882357793" target="_blank" rel="noopener nofollow ugc">github.com/NVIDIA/nccl</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/NVIDIA/nccl/issues/1141#issuecomment-1882357793" target="_blank" rel="noopener nofollow ugc">torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, remote process exited or there was a network error, NCCL version 2.18.6</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2024-01-08" data-time="09:03:28" data-timezone="UTC">09:03AM - 08 Jan 24 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/Rainbowman0" target="_blank" rel="noopener nofollow ugc">
          <img alt="Rainbowman0" src="https://avatars.githubusercontent.com/u/60497059?v=4" class="onebox-avatar-inline" width="20" height="20">
          Rainbowman0
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Hi, I successfully ran the ['cifar10_deepspeed.py' ](https://www.deepspeed.ai/tu<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">torials/cifar-10/)example on a single node (2xNVIDIA 3090). Now I want to run the same program on multi-nodes (2 nodes each have 2 3090s.). I refer to the example [here ](https://nlp.stanford.edu/mistral/tutorials/deepspeed.html)to run my program.

my hostfile:
```bash
192.168.3.100 slots=2
192.168.3.101 slots=2
```

then I ran the command:
```bash
deepspeed --num_gpus 2 --num_nodes 2 --master_addr 192.168.3.100 --hostfile hostfile cifar10_deepspeed.py --deepspeed_config ds_config.json
```

I also ran these commands on both 192.168.3.100 and 192.168.3.101:
```bash
export NCCL_IB_DISABLE=1
export NCCL_SOCKET_IFNAME=eno1
export NCCL_P2P_DISABLE=1
export NCCL_DEBUG=INFO
```
And I met the error:
```bash
[2024-01-08 16:13:55,768] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[2024-01-08 16:13:56,462] [INFO] [multinode_runner.py:80:get_cmd] Running on the following workers: 192.168.3.100,192.168.3.101
[2024-01-08 16:13:56,463] [INFO] [runner.py:571:main] cmd = pdsh -S -f 1024 -w 192.168.3.100,192.168.3.101 source /root/SYH/GithubCode/DeepSpeedExamples/training/cifar/setup_env1.sh export NCCL_SOCKET_IFNAME=eno1; export PYTHONPATH=/root/SYH/GithubCode/DeepSpeedExamples/training/cifar;  cd /root/SYH/GithubCode/DeepSpeedExamples/training/cifar; /root/anaconda3/envs/DS/bin/python -u -m deepspeed.launcher.launch --world_info=eyIxOTIuMTY4LjMuMTAwIjogWzAsIDFdLCAiMTkyLjE2OC4zLjEwMSI6IFswLCAxXX0= --node_rank=%n --master_addr=192.168.3.100 --master_port=29500 cifar10_deepspeed.py --deepspeed_config ds_config.json
192.168.3.100: [2024-01-08 16:13:59,770] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.101: [2024-01-08 16:13:59,842] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.100: [2024-01-08 16:14:00,061] [INFO] [launch.py:145:main] WORLD INFO DICT: {'192.168.3.100': [0, 1], '192.168.3.101': [0, 1]}
192.168.3.100: [2024-01-08 16:14:00,061] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=2, node_rank=0
192.168.3.100: [2024-01-08 16:14:00,061] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(&lt;class 'list'&gt;, {'192.168.3.100': [0, 1], '192.168.3.101': [2, 3]})
192.168.3.100: [2024-01-08 16:14:00,061] [INFO] [launch.py:163:main] dist_world_size=4
192.168.3.100: [2024-01-08 16:14:00,061] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1
192.168.3.101: [2024-01-08 16:14:00,102] [INFO] [launch.py:145:main] WORLD INFO DICT: {'192.168.3.100': [0, 1], '192.168.3.101': [0, 1]}
192.168.3.101: [2024-01-08 16:14:00,102] [INFO] [launch.py:151:main] nnodes=2, num_local_procs=2, node_rank=1
192.168.3.101: [2024-01-08 16:14:00,103] [INFO] [launch.py:162:main] global_rank_mapping=defaultdict(&lt;class 'list'&gt;, {'192.168.3.100': [0, 1], '192.168.3.101': [2, 3]})
192.168.3.101: [2024-01-08 16:14:00,103] [INFO] [launch.py:163:main] dist_world_size=4
192.168.3.101: [2024-01-08 16:14:00,103] [INFO] [launch.py:165:main] Setting CUDA_VISIBLE_DEVICES=0,1
192.168.3.100: [2024-01-08 16:14:02,778] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.101: [2024-01-08 16:14:02,794] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.101: [2024-01-08 16:14:02,802] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.100: [2024-01-08 16:14:02,807] [INFO] [real_accelerator.py:161:get_accelerator] Setting ds_accelerator to cuda (auto detect)
192.168.3.100: [2024-01-08 16:14:02,998] [INFO] [comm.py:637:init_distributed] cdb=None
192.168.3.100: [2024-01-08 16:14:03,032] [INFO] [comm.py:637:init_distributed] cdb=None
192.168.3.100: [2024-01-08 16:14:03,032] [INFO] [comm.py:668:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
192.168.3.101: [2024-01-08 16:14:03,084] [INFO] [comm.py:637:init_distributed] cdb=None
192.168.3.101: [2024-01-08 16:14:03,110] [INFO] [comm.py:637:init_distributed] cdb=None
192.168.3.100: Files already downloaded and verified
192.168.3.101: Traceback (most recent call last):
192.168.3.101:   File "/root/SYH/GithubCode/DeepSpeedExamples/training/cifar/cifar10_deepspeed.py", line 133, in &lt;module&gt;
192.168.3.101:     torch.distributed.barrier()
192.168.3.101:   File "/root/anaconda3/envs/DS/lib/python3.9/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
192.168.3.101:     return func(*args, **kwargs)
192.168.3.101:   File "/root/anaconda3/envs/DS/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
192.168.3.101:     work = default_pg.barrier(opts=opts)
192.168.3.101: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, remote process exited or there was a network error, NCCL version 2.18.6
192.168.3.101: ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
192.168.3.101: Last error:
192.168.3.101: socketProgressOpt: Call to recv from 192.168.3.101&lt;47981&gt; failed : Broken pipe
192.168.3.100: Traceback (most recent call last):
192.168.3.100:   File "/root/SYH/GithubCode/DeepSpeedExamples/training/cifar/cifar10_deepspeed.py", line 133, in &lt;module&gt;
192.168.3.100:     torch.distributed.barrier()
192.168.3.100:   File "/root/anaconda3/envs/DS/lib/python3.9/site-packages/torch/distributed/c10d_logger.py", line 47, in wrapper
192.168.3.100:     return func(*args, **kwargs)
192.168.3.100:   File "/root/anaconda3/envs/DS/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 3696, in barrier
192.168.3.100:     work = default_pg.barrier(opts=opts)
192.168.3.100: torch.distributed.DistBackendError: NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1333, remote process exited or there was a network error, NCCL version 2.18.6
192.168.3.100: ncclRemoteError: A call failed possibly due to a network error or a remote process exiting prematurely.
192.168.3.100: Last error:
192.168.3.100: socketProgressOpt: Call to recv from 192.168.3.100&lt;54189&gt; failed : Broken pipe
192.168.3.100: [2024-01-08 16:14:07,091] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3422546
192.168.3.101: [2024-01-08 16:14:07,133] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 1871110
192.168.3.100: [2024-01-08 16:14:07,256] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 3422547
192.168.3.100: [2024-01-08 16:14:07,257] [ERROR] [launch.py:321:sigkill_handler] ['/root/anaconda3/envs/DS/bin/python', '-u', 'cifar10_deepspeed.py', '--local_rank=1', '--deepspeed_config', 'ds_config.json'] exits with return code = 1
192.168.3.101: [2024-01-08 16:14:07,299] [INFO] [launch.py:315:sigkill_handler] Killing subprocess 1871111
192.168.3.101: [2024-01-08 16:14:07,299] [ERROR] [launch.py:321:sigkill_handler] ['/root/anaconda3/envs/DS/bin/python', '-u', 'cifar10_deepspeed.py', '--local_rank=1', '--deepspeed_config', 'ds_config.json'] exits with return code = 1
pdsh@pai-worker1: 192.168.3.100: ssh exited with exit code 1
pdsh@pai-worker1: 192.168.3.101: ssh exited with exit code 1
```

It looks like something wrong with NCCL? I'm not sure how to fix it. Do you have any suggestions? :)</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


nishantchanduka: <p>I am facing this error. The jobs are running on one node. But face the error every time I choose more than 2 GPUs to work. If I restrict working with 2 GPUs it works fine. And the issue is not restricted to one node. I tried launching job of different node still I face the same.</p>
<p>[2024-07-25T01:10:59.170447Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO Channel 20/0 : 3[3] → 2[2] via P2P/IPC<br>
[2024-07-25T01:10:59.170470Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO Channel 21/0 : 3[3] → 2[2] via P2P/IPC<br>
[2024-07-25T01:10:59.170492Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO Channel 22/0 : 3[3] → 2[2] via P2P/IPC<br>
[2024-07-25T01:10:59.170513Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO Channel 23/0 : 3[3] → 2[2] via P2P/IPC<br>
[2024-07-25T01:10:59.170535Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO Connected all trees<br>
[2024-07-25T01:10:59.170558Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO NVLS comm 0x55cf72020af0 headRank 3 nHeads 4 buffSize 4194304 memSize 2097152 nvlsPerRankSize 201326592 nvlsTotalSize 805306368<br>
[2024-07-25T01:10:59.170579Z] b9f5df8d [rank=3] ||<br>
[2024-07-25T01:10:59.170602Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] transport/nvls.cc:165 NCCL WARN Cuda failure ‘system not yet initialized’<br>
[2024-07-25T01:10:59.170624Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO transport/nvls.cc:324 → 1<br>
[2024-07-25T01:10:59.170646Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO init.cc:1093 → 1<br>
[2024-07-25T01:10:59.170668Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO init.cc:1358 → 1<br>
[2024-07-25T01:10:59.170690Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:1707 [3] NCCL INFO group.cc:65 → 1 [Async thread]<br>
[2024-07-25T01:10:59.170711Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:816 [3] NCCL INFO group.cc:406 → 1<br>
[2024-07-25T01:10:59.170733Z] b9f5df8d [rank=3] || 53ab7896e3a7:816:816 [3] NCCL INFO group.cc:96 → 1</p>
<p>[2024-07-25T01:10:59.198876Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py”, line 566, in from_pretrained<br>
[2024-07-25T01:10:59.198881Z] b9f5df8d [rank=2] ||     return model_class.from_pretrained(<br>
[2024-07-25T01:10:59.198895Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py”, line 3462, in from_pretrained<br>
[2024-07-25T01:10:59.198900Z] b9f5df8d [rank=2] ||     model = cls(config, *model_args, **model_kwargs)<br>
[2024-07-25T01:10:59.198905Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/partition_parameters.py”, line 459, in wrapper<br>
[2024-07-25T01:10:59.198910Z] b9f5df8d [rank=2] ||     f(module, *args, **kwargs)<br>
[2024-07-25T01:10:59.198917Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py”, line 1109, in <strong>init</strong><br>
[2024-07-25T01:10:59.198922Z] b9f5df8d [rank=2] ||     self.model = LlamaModel(config)<br>
[2024-07-25T01:10:59.198928Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/partition_parameters.py”, line 459, in wrapper<br>
[2024-07-25T01:10:59.198947Z] b9f5df8d [rank=2] ||     f(module, *args, **kwargs)<br>
[2024-07-25T01:10:59.198957Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/transformers/models/llama/modeling_llama.py”, line 954, in <strong>init</strong><br>
[2024-07-25T01:10:59.198964Z] b9f5df8d [rank=2] ||     self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size, self.padding_idx)<br>
[2024-07-25T01:10:59.198983Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/partition_parameters.py”, line 466, in wrapper<br>
[2024-07-25T01:10:59.198992Z] b9f5df8d [rank=2] ||     self._post_init_method(module)<br>
[2024-07-25T01:10:59.199262Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/partition_parameters.py”, line 1000, in _post_init_method<br>
[2024-07-25T01:10:59.199275Z] b9f5df8d [rank=2] ||     self._zero_init_param(param)<br>
[2024-07-25T01:10:59.199282Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/runtime/zero/partition_parameters.py”, line 956, in _zero_init_param<br>
[2024-07-25T01:10:59.199290Z] b9f5df8d [rank=2] ||     dist.broadcast(param, 0, self.get_dp_process_group())<br>
[2024-07-25T01:10:59.199295Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py”, line 117, in log_wrapper<br>
[2024-07-25T01:10:59.199301Z] b9f5df8d [rank=2] ||     return func(*args, **kwargs)<br>
[2024-07-25T01:10:59.199306Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/comm/comm.py”, line 224, in broadcast<br>
[2024-07-25T01:10:59.199312Z] b9f5df8d [rank=2] ||     return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)<br>
[2024-07-25T01:10:59.199317Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/deepspeed/comm/torch.py”, line 196, in broadcast<br>
[2024-07-25T01:10:59.199322Z] b9f5df8d [rank=2] ||     return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)<br>
[2024-07-25T01:10:59.199327Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/torch/distributed/c10d_logger.py”, line 47, in wrapper<br>
[2024-07-25T01:10:59.199332Z] b9f5df8d [rank=2] ||     return func(*args, **kwargs)<br>
[2024-07-25T01:10:59.199339Z] b9f5df8d [rank=2] ||   File “/usr/local/lib/python3.10/dist-packages/torch/distributed/distributed_c10d.py”, line 1900, in broadcast<br>
[2024-07-25T01:10:59.199345Z] b9f5df8d [rank=2] ||     work = default_pg.broadcast([tensor], opts)<br>
[2024-07-25T01:10:59.199351Z] b9f5df8d [rank=2] || torch.distributed.DistBackendError: NCCL error in: /opt/pytorch/pytorch/torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1207, unhandled cuda error (run with NCCL_DEBUG=INFO for details), NCCL version 2.18.5<br>
[2024-07-25T01:10:59.199356Z] b9f5df8d [rank=2] || ncclUnhandledCudaError: Call to CUDA function failed.<br>
[2024-07-25T01:10:59.199361Z] b9f5df8d [rank=2] || Last error:<br>
[2024-07-25T01:10:59.199366Z] b9f5df8d [rank=2] || Cuda failure ‘system not yet initialized’</p>

ptrblck: <p>Could you check if your system runs NVLink/NVSwitch fabric manager? If not, you might want to install it.</p>