Spawned Processes with DDP

username: kleingeo
cooked: <p>When I run a model with DDP with 4 spawned processes and each process taking a GPU, I notice that there are a lot of spawned processes with <code>nvidia-smi</code> on the main process. Why are there so many additional processes spawned? I am also using <code>torch=1.4.0</code> and <code>CUDA=10.1.243</code> which is installed through my conda environment.</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bca424a1ea29ec21b71c03eac5dfb4f48ba3fa2.png" data-download-href="https://discuss.pytorch.org/uploads/default/9bca424a1ea29ec21b71c03eac5dfb4f48ba3fa2" title="gpu_ddp"><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/b/9bca424a1ea29ec21b71c03eac5dfb4f48ba3fa2.png" alt="gpu_ddp" data-base62-sha1="meblqTLIyFMMOQ9HHaajRTQSGau" width="500" height="500" data-dominant-color="2E2E2E"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu_ddp</span><span class="informations">642×642 11.6 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: iffiX
cooked: <p>See discussion <a href="https://discuss.pytorch.org/t/torch-distributed-barrier-used-in-multi-node-distributed-data-parallel-training/89711/24" class="inline-onebox">`torch.distributed.barrier` used in multi-node distributed data-parallel training</a></p>
<p>For more detailed explainations, cc <a class="mention" href="/u/mrshenli">@mrshenli</a></p>

username: kleingeo
cooked: <aside class="quote no-group" data-username="iffiX" data-post="2" data-topic="90526">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/iffix/48/24443_2.png" class="avatar"> iffiX:</div>
<blockquote>
<p>For more detailed explainations, c</p>
</blockquote>
</aside>
<p>I took a look at that, but I am a little confused as it still. Is it a back-end issue, or is it normal behaviour?</p>

username: iffiX
cooked: <p>should be normal，depending on the backend</p>

username: kleingeo
cooked: <p>It seems to happen with both nccl and gloo backends.</p>

username: iffiX
cooked: <p>Maybe ask <a class="mention" href="/u/mrshenli">@mrshenli</a>, I am not fully clear about their internal implementations.</p>

username: kleingeo
cooked: <p>So I played around a little more by breaking my pipeline to a very simple training loop, just to ensure I wasn’t doing anything wrong. I then cloned my conda environment and updated the torch and cuda to the versions listed on the Getting Started page (the most up-to-date) and it seems to fix the issue. So I’m not sure if it was a cuda, nccl (in the cudatoolkit), or the updated torch, but updating does fix it.</p>

username: mrshenli
cooked: <p>Those (~500MB CUDA memory consumption) look like CUDA context. It seems all processes somehow used <code>CUDA:0</code> somewhere. It could be caused by 3rd-party libraries or calls like <code>torch.cuda.empty_cache()</code>. If you don’t want to debug the root cause of it, you can avoid it by setting <code>CUDA_VISIBLE_DEVICES</code> env var to make sure that each process only sees one GPU.</p>

username: kleingeo
cooked: <p>I updated both my torch and cudatoolkit (with conda) to the newest versions which seemed to fix the problem. I’m not sure if it was cuda, torch or how torch interacted with cuda, but updating seems to have fixed the bug. I’m not sure if this improves anything though, but the extra processes don’t show up anymore.</p>