DDP (via Lightning/Fabric) training hang with 100% GPU utilization

Clion: <p>Ok, here’s the problem. First of all, this is a Fabric (/Lightning) problem with multi-GPU training. My code hangs upon reaching this line:</p>
<pre><code class="lang-auto">aNet,opt = fabric.setup( aNet,opt )
</code></pre>
<p>where aNet is a custom model, subclassing nn.Module as per the usual, and opt is defined thusly:</p>
<pre><code class="lang-auto">opt = torch.optim.Adam( 
    aNet.parameters(), lr=lRate, eps=1e-08, foreach=True )
</code></pre>
<p>The following is what nvitop is showing when the <code>fabric.setup</code> line executes:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/4/94ac287ac4580883e00923f0ca8faf0b822ed6cf.png" data-download-href="https://discuss.pytorch.org/uploads/default/94ac287ac4580883e00923f0ca8faf0b822ed6cf" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94ac287ac4580883e00923f0ca8faf0b822ed6cf_2_690x171.png" alt="image" data-base62-sha1="lddvMPPyucVqgA8FVi5IJ9Bkm6P" width="690" height="171" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94ac287ac4580883e00923f0ca8faf0b822ed6cf_2_690x171.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94ac287ac4580883e00923f0ca8faf0b822ed6cf_2_1035x256.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/4/94ac287ac4580883e00923f0ca8faf0b822ed6cf_2_1380x342.png 2x" data-dominant-color="0E1B07"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1901×473 106 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>These lines are just before the one that causes the freeze, and they execute fine (don’t worry about what exactly the callback thing is - it’s not relevant here) :</p>
<pre><code class="lang-auto">nGPU = torch.cuda.device_count()
fabric = Fabric( 
    accelerator='cuda', devices=nGPU, strategy="ddp", callbacks=[TM] )
fabric.launch()
</code></pre>
<p>Conda gives me the following when queried about currently installed version of cuda and pytorch:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de48ea9b46994b3074fa31a6d0c4209d64492ff.png" data-download-href="https://discuss.pytorch.org/uploads/default/4de48ea9b46994b3074fa31a6d0c4209d64492ff" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4de48ea9b46994b3074fa31a6d0c4209d64492ff_2_690x60.png" alt="image" data-base62-sha1="b74rI0rr246mCWxND1ePCSdliOP" width="690" height="60" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/d/4de48ea9b46994b3074fa31a6d0c4209d64492ff_2_690x60.png, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de48ea9b46994b3074fa31a6d0c4209d64492ff.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/d/4de48ea9b46994b3074fa31a6d0c4209d64492ff.png 2x" data-dominant-color="001500"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">916×80 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>I’m on pretty much the newest everything because I just completely re-created this env from the bottom up to address this issue, as I thought it was simply because my previous one wasn’t up to date. This is what I was previously running, which was giving me the exact same issue:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/6/36e2c04488db4832d633452b9d8d0485d180c4d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/36e2c04488db4832d633452b9d8d0485d180c4d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36e2c04488db4832d633452b9d8d0485d180c4d1_2_690x67.png" alt="image" data-base62-sha1="7PxAUfI2AYva3kk5Y8v4DU78l6p" width="690" height="67" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/3/6/36e2c04488db4832d633452b9d8d0485d180c4d1_2_690x67.png, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36e2c04488db4832d633452b9d8d0485d180c4d1.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/3/6/36e2c04488db4832d633452b9d8d0485d180c4d1.png 2x" data-dominant-color="011400"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">825×81 16.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What’s odd is that this only happens when &gt;1 GPUs are present, and only with some GPUs. As you can see from the first image, this machine is running two L40s. If I run only one L40, training proceeds just fine. So far, I’ve confirmed this problem to happen with the following GPUs: L40, RTX 6000 Ada, and A100 (PCIe and SXM) - all of these work totally fine if I’m only using one, but when &gt;1 of any of these are present, all of them immediately jump to maxed-out GPU utilization when the offending line is reached. I’ve also confirmed that this issue doesn’t occur with: RTX 4090, RTX A6000, or RTX 3090, regardless of how many of them I’m running in parallel.</p>
<p>Any ideas? Happy to provide any more information needed or test on other machines.</p>

Clion: <p>To future readers:</p>
<p>Solved this problem thanks to this: <a href="https://discuss.pytorch.org/t/single-machine-ddp-issue-on-a6000-gpu/134869" class="inline-onebox">Single Machine DDP Issue on A6000 GPU</a></p>
<p>The tl;dr is that, for me, setting this env var was enough to fix the problem entirely: <code>NCCL_P2P_DISABLE=1</code></p>
<p>If you are using conda you probably want to export this var to the activation script which is run every time you start an env, like so:</p>
<pre><code class="lang-auto">echo 'export NCCL_P2P_DISABLE=1' &gt; $CONDA_PREFIX/etc/conda/activate.d/env_vars.sh
</code></pre>
<p>After doing the above, problem should be fixed for you forever (“forever”)…at least if you’re as lucky as I was (:</p>

Erotemic: <p>I have a similar issue on a two RTX 3090’s on a single machine, however, the NCCL_P2P_DISABLE=1 environ does not fix the issue for me. The issue happens only when I have a custom callback enabled that plots the first few batches in each epoch. I have callback guarded so it only executes on rank 0. I can train for awhile, but then it locks up.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc070adc1088e95129d5217360ffdc9b8874c9ae.png" data-download-href="https://discuss.pytorch.org/uploads/default/cc070adc1088e95129d5217360ffdc9b8874c9ae" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc070adc1088e95129d5217360ffdc9b8874c9ae_2_690x77.png" alt="image" data-base62-sha1="t6UuwlXtgR6yJzo2wXoT5GpomgC" width="690" height="77" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc070adc1088e95129d5217360ffdc9b8874c9ae_2_690x77.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/c/cc070adc1088e95129d5217360ffdc9b8874c9ae_2_1035x115.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/c/cc070adc1088e95129d5217360ffdc9b8874c9ae.png 2x" data-dominant-color="323234"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1241×139 15.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>100% utilization should be using way more than 148/370 Watts.</p>
<p>Disabling the callback does work to fix it, but I’d be interested in more info on why this can happen in the first place.</p>