Distributed stuck in first call to barrier

tom: <p>Hi I am stuck and could use a hint on distributed.</p>
<p>I’m trying to run a distributed model on a two-GPU machine. Until recently, that seemed to work, but it has stopped. Now it is hanging at the first barrier call.<br>
The NCCL_DEBUG (below with a few funny debug prints) shows “Init COMPLETE” but then stops (will timeout eventually).</p>
<p>Any hint would be appreciated!</p>
<p>The full code in question is this (called from running the test_fsdp.py from the same directory.)</p>
<aside class="onebox githubblob" data-onebox-src="https://github.com/Lightning-AI/lightning-thunder/blob/fa307a5656d2b7000a07a489f3d434aa7ba8cfd6/thunder/tests/distributed/helper.py#L122-L133">
  <header class="source">

      <a href="https://github.com/Lightning-AI/lightning-thunder/blob/fa307a5656d2b7000a07a489f3d434aa7ba8cfd6/thunder/tests/distributed/helper.py#L122-L133" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/Lightning-AI/lightning-thunder/blob/fa307a5656d2b7000a07a489f3d434aa7ba8cfd6/thunder/tests/distributed/helper.py#L122-L133" target="_blank" rel="noopener nofollow ugc">Lightning-AI/lightning-thunder/blob/fa307a5656d2b7000a07a489f3d434aa7ba8cfd6/thunder/tests/distributed/helper.py#L122-L133</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="122" style="counter-reset: li-counter 121 ;">
          <li>torch.distributed.init_process_group(</li>
          <li>    init_method=self.init_method,</li>
          <li>    backend=self.DISTRIBUTED_BACKEND,</li>
          <li>    world_size=self.world_size,</li>
          <li>    rank=self.rank,</li>
          <li>)</li>
          <li></li>
          <li>local_rank = self.rank % torch.cuda.device_count()</li>
          <li>torch.cuda.set_device(local_rank)</li>
          <li>os.environ["LOCAL_RANK"] = str(local_rank)</li>
          <li></li>
          <li>torch.distributed.barrier()</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">thunder/tests/distributed/test_fsdp.py::FSDPTest::test_fsdp_broadcast_from #####initializing with file:///tmp/tmpil2bmukt #### 0 ###### 2 ##### /tmp/tmpil2bmukt
#####initializing with file:///tmp/tmpil2bmukt #### 1 ###### 2 ##### /tmp/tmpil2bmukt
####returned init
####returned init
###call barrier rank 0
###call barrier rank 1
mackay:2172718:2172718 [0] NCCL INFO Bootstrap : Using enp4s0:192.168.1.1&lt;0&gt;
mackay:2172718:2172718 [0] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
mackay:2172718:2172718 [0] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
mackay:2172718:2172718 [0] NCCL INFO NET/Plugin: Using internal network plugin.
mackay:2172718:2172718 [0] NCCL INFO cudaDriverVersion 12030
NCCL version 2.21.5+cuda12.2
mackay:2172719:2172719 [1] NCCL INFO cudaDriverVersion 12030
mackay:2172719:2172719 [1] NCCL INFO Bootstrap : Using enp4s0:192.168.1.1&lt;0&gt;
mackay:2172719:2172719 [1] NCCL INFO NET/Plugin: No plugin found (libnccl-net.so)
mackay:2172719:2172719 [1] NCCL INFO NET/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-net.so
mackay:2172719:2172719 [1] NCCL INFO NET/Plugin: Using internal network plugin.
mackay:2172718:2172799 [0] NCCL INFO NET/IB : No device found.
mackay:2172718:2172799 [0] NCCL INFO NET/Socket : Using [0]enp4s0:192.168.1.1&lt;0&gt; [1]enp5s0:192.168.178.37&lt;0&gt;
mackay:2172718:2172799 [0] NCCL INFO Using non-device net plugin version 0
mackay:2172718:2172799 [0] NCCL INFO Using network Socket
mackay:2172719:2172800 [1] NCCL INFO NET/IB : No device found.
mackay:2172719:2172800 [1] NCCL INFO NET/Socket : Using [0]enp4s0:192.168.1.1&lt;0&gt; [1]enp5s0:192.168.178.37&lt;0&gt;
mackay:2172719:2172800 [1] NCCL INFO Using non-device net plugin version 0
mackay:2172719:2172800 [1] NCCL INFO Using network Socket
mackay:2172718:2172799 [0] NCCL INFO ncclCommInitRank comm 0xa761300 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a000 commId 0x42b80bb4bcb10b00 - Init START
mackay:2172719:2172800 [1] NCCL INFO ncclCommInitRank comm 0x94f4b30 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b000 commId 0x42b80bb4bcb10b00 - Init START
mackay:2172719:2172800 [1] NCCL INFO comm 0x94f4b30 rank 1 nRanks 2 nNodes 1 localRanks 2 localRank 1 MNNVL 0
mackay:2172718:2172799 [0] NCCL INFO comm 0xa761300 rank 0 nRanks 2 nNodes 1 localRanks 2 localRank 0 MNNVL 0
mackay:2172719:2172800 [1] NCCL INFO Trees [0] -1/-1/-1-&gt;1-&gt;0 [1] -1/-1/-1-&gt;1-&gt;0
mackay:2172718:2172799 [0] NCCL INFO Channel 00/02 :    0   1
mackay:2172719:2172800 [1] NCCL INFO P2P Chunksize set to 131072
mackay:2172718:2172799 [0] NCCL INFO Channel 01/02 :    0   1
mackay:2172718:2172799 [0] NCCL INFO Trees [0] 1/-1/-1-&gt;0-&gt;-1 [1] 1/-1/-1-&gt;0-&gt;-1
mackay:2172718:2172799 [0] NCCL INFO P2P Chunksize set to 131072
mackay:2172718:2172799 [0] NCCL INFO Channel 00/0 : 0[0] -&gt; 1[1] via P2P/IPC
mackay:2172719:2172800 [1] NCCL INFO Channel 00/0 : 1[1] -&gt; 0[0] via P2P/IPC
mackay:2172719:2172800 [1] NCCL INFO Channel 01/0 : 1[1] -&gt; 0[0] via P2P/IPC
mackay:2172718:2172799 [0] NCCL INFO Channel 01/0 : 0[0] -&gt; 1[1] via P2P/IPC
mackay:2172719:2172800 [1] NCCL INFO Connected all rings
mackay:2172719:2172800 [1] NCCL INFO Connected all trees
mackay:2172719:2172800 [1] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
mackay:2172718:2172799 [0] NCCL INFO Connected all rings
mackay:2172719:2172800 [1] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mackay:2172718:2172799 [0] NCCL INFO Connected all trees
mackay:2172718:2172799 [0] NCCL INFO threadThresholds 8/8/64 | 16/8/64 | 512 | 512
mackay:2172718:2172799 [0] NCCL INFO 2 coll channels, 2 collnet channels, 0 nvls channels, 2 p2p channels, 2 p2p channels per peer
mackay:2172719:2172800 [1] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
mackay:2172719:2172800 [1] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
mackay:2172719:2172800 [1] NCCL INFO ncclCommInitRank comm 0x94f4b30 rank 1 nranks 2 cudaDev 1 nvmlDev 1 busId b000 commId 0x42b80bb4bcb10b00 - Init COMPLETE
mackay:2172718:2172799 [0] NCCL INFO TUNER/Plugin: Plugin load returned 2 : libnccl-net.so: cannot open shared object file: No such file or directory : when loading libnccl-tuner.so
mackay:2172718:2172799 [0] NCCL INFO TUNER/Plugin: Using internal tuner plugin.
mackay:2172718:2172799 [0] NCCL INFO ncclCommInitRank comm 0xa761300 rank 0 nranks 2 cudaDev 0 nvmlDev 0 busId a000 commId 0x42b80bb4bcb10b00 - Init COMPLETE

</code></pre>

agu: <p>Do you have any sense of when it stopped working in case it helps us bisect the issue? Were there any changes in NCCL version?</p>

tom: <p>Unfortunately, I don’t know. It was some time during the last two months, and I’m running a self-compiled PyTorch, so this would give quite a bisection task. <img src="https://discuss.pytorch.org/images/emoji/apple/frowning.png?v=12" title=":frowning:" class="emoji" alt=":frowning:" loading="lazy" width="20" height="20"><br>
I wonder if there is a way to discern what should be happening (like “Waiting for  to happen at network address ”) or so in order to know  <strong>what</strong> it is that I need to look at why it’s not happening…</p>

tom: <p>So what I did was grab a docker container in which I knew distributed worked and when that failed, it was clear that it was the driver. Downgrading the nvidia driver helped.<br>
Of course, it would be nice if there was a proper error message somewhere if NCCL didn’t like my driver, but I guess that’s not a PyTorch thing.</p>