Distributed Data Parallel Training Extra GPU n-1 process on n GPU process job

vx443: <p>I’m using DDP for training (using DDP wrapper around the model) and when I spawn n jobs I see the process on with nvidia-smi/gpustat but then there are (n-1) other GPU processes that are also created. I’m guessing that these are for communication between gpu0 and the other processes. I don’t see anything in the documentation or this forum about that. Below is output from gpustat for a 2 GPU job. The 2 GPU process using memory 9763 are the model and then there is 565M size process on gpu0. The latter process seems to be about the same size for any model.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05.png" data-download-href="https://discuss.pytorch.org/uploads/default/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05" title="ddp_gpustat"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05_2_690x126.png" alt="ddp_gpustat" data-base62-sha1="j3Gw49cXA27NZoz0jIStZjlqdRr" width="690" height="126" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05_2_690x126.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05_2_1035x189.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/8593f70cef2f3311cf89faa0ac7e7606c1fa0b05.png 2x" data-dominant-color="0A0C09"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">ddp_gpustat</span><span class="informations">1094×200 34.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>So is this normal or is it an incorrect setup?<br>
Thanks.</p>

mrshenli: <p>Could you please check the process ids? Each DDP process is supposed to only work on one device. Suppose the expected case is process rank 0 on <code>cuda:0</code> and process rank1 on <code>cuda:1</code>. It’s likely that somehow process rank1 also created a CUDA context on <code>cuda:0</code>. The size of <code>565M</code> also looks like a CUDA context.</p>
<p>If it’s indeed the case, there are a few actions can help avoid such problem:</p>
<ol>
<li>Run <code>torch.cuda.set_device(rank)</code> on each process to properly set the current device before running any CUDA ops.</li>
<li>If 1 still does not solve the problem, you can set <code>CUDA_VISIBLE_DEVICES</code> env var to make sure that process 0 only sees gpu0 and process 1 only sees gpu1.</li>
</ol>

hrushikesh198: <p>I was seeing the same issue with multi gpu training(DDP strategy). n-1 extra processes on each gpu holding ~1400MiB on A100 Gpus.</p>
<p>I zeroed down the issue to <code>pytorch-lightning's Trainer() flag auto_select_gpus=True</code>. Setting it to False solve the issue. Now I can train with larger batch size to fully utilize the gpu memory when I want to.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/e/c/ec66aa1466b9c263e2fbc57f88f648517eb5eba3.png" data-download-href="https://discuss.pytorch.org/uploads/default/ec66aa1466b9c263e2fbc57f88f648517eb5eba3" title="Screen Shot 2022-05-24 at 12.33.26 PM"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec66aa1466b9c263e2fbc57f88f648517eb5eba3_2_409x500.png" alt="Screen Shot 2022-05-24 at 12.33.26 PM" data-base62-sha1="xJiDHOYrdjsPYWADpAF5OrUzN3t" width="409" height="500" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec66aa1466b9c263e2fbc57f88f648517eb5eba3_2_409x500.png, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec66aa1466b9c263e2fbc57f88f648517eb5eba3_2_613x750.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/e/c/ec66aa1466b9c263e2fbc57f88f648517eb5eba3_2_818x1000.png 2x" data-dominant-color="0F0F0F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screen Shot 2022-05-24 at 12.33.26 PM</span><span class="informations">1432×1748 356 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>