Python processes and GPU usage during distributed training

username: 742617000027
cooked: <p>Hey all!</p>
<p>I noticed that after the first epoch of training a custom DDP model (1 node, 2 GPUs), two new GPU memory usage entries pop up in <code>nvidia-smi</code>, one for each GPU. A memory usage of around 800 MB is reported for these. Curiously, they have the same PID as the ‘main’ process running on the opposite GPU (see the color-coded screenshot below). Is this expected behavior? It’s the only case of training a model with data parallelism where I noticed it.<br>
I’ve tried using several monitoring tools besides <code>nvidia-smi</code>, e.g. <code>GPUtil.showUtilization()</code>, <code>torch.cuda.memory_allocated()</code>, <a href="https://docs.fast.ai/dev/gpu.html#pytorch-tensor-memory-tracking" rel="noopener nofollow ugc">printing tensors in use</a>, but none of these are descriptive enough for me to pinpoint what exactly is going on.</p>
<p>Any help, suggestions or input would be greatly appreciated!</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bc6647f39b103c5b83ce6e0853affc2737f013.png" data-download-href="https://discuss.pytorch.org/uploads/default/c7bc6647f39b103c5b83ce6e0853affc2737f013" title="torch_colored"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bc6647f39b103c5b83ce6e0853affc2737f013_2_690x425.png" alt="torch_colored" data-base62-sha1="suWEXYaAULdaJkr4rSotud5aCNZ" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bc6647f39b103c5b83ce6e0853affc2737f013_2_690x425.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/7/c7bc6647f39b103c5b83ce6e0853affc2737f013_2_1035x637.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/7/c7bc6647f39b103c5b83ce6e0853affc2737f013.png 2x" data-dominant-color="2B2C2A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">torch_colored</span><span class="informations">1050×648 373 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: mrshenli
cooked: <blockquote>
<p>after the first epoch of training a custom DDP model (1 node, 2 GPUs), two new GPU memory usage entries pop up in  <code>nvidia-smi</code></p>
</blockquote>
<p>You mean those two entries was not there during the 1 epoch, and only appears after that? What was executed after the first epoch? How did you initialize DDP? Did you specify <code>device_ids</code> or set <code>CUDA_VISIBLE_DEVICES</code> for the two DDP instances? Looks like both processes are operating on two GPUs. They should have exclusively run on a different GPU.</p>

username: 742617000027
cooked: <blockquote>
<p>You mean those two entries was not there during the 1 epoch, and only appears after that?</p>
</blockquote>
<p>Yes, that’s correct.</p>
<blockquote>
<p>What was executed after the first epoch?</p>
</blockquote>
<p>After each epoch of training, evaluation is run on GPU 0.</p>
<blockquote>
<p>How did you initialize DDP?</p>
</blockquote>
<p>With <code>torch.distributed.launch --nproc_per_node=2</code> and the following:</p>
<pre><code class="lang-auto">torch.cuda.set_device(args.local_rank)
torch.distributed.init_process_group(backend='nccl', init_method='env://')
model.cuda(args.local_rank)
model =  torch.nn.parallel.DistributedDataParallel(model, 
                                                   device_ids=[args.local_rank],
                                                   output_device=args.local_rank,
                                                   find_unused_parameters=True)
</code></pre>
<p>(<code>args.local_rank</code> is set automatically by <code>torch.distributed.launch</code>)</p>
<p>Appreciate the response!</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="742617000027" data-post="3" data-topic="71735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/742617000027/48/21226_2.png" class="avatar"> 742617000027:</div>
<blockquote>
<p>After each epoch of training, evaluation is run on GPU 0.</p>
</blockquote>
</aside>
<p>Does this occur on all processes or just rank 0?</p>

username: 742617000027
cooked: <p>Evaluation is gated by an <code>if</code> clause:</p>
<pre><code class="lang-auto">if args.local_rank == 0:
    evaluate(params)
</code></pre>
<p>Is that what you were asking?</p>

username: mrshenli
cooked: <aside class="quote no-group" data-username="742617000027" data-post="5" data-topic="71735">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/742617000027/48/21226_2.png" class="avatar"> 742617000027:</div>
<blockquote>
<p>Is that what you were asking?</p>
</blockquote>
</aside>
<p>Yes.</p>
<pre data-code-wrap="python"><code class="lang-python">model = torch.distributed.nn.parallel.DistributedDataParallel(model)
</code></pre>
<p>The line above might be the reason. IIUC, <code>torch.cuda.set_device(args.local_rank)</code> only sets the default device, but DDP process can still see two GPUS. By default DDP would try to replicate models on all visible GPUs.</p>
<p>Could you please try use the following to construct DDP?</p>
<pre><code class="lang-auto">    model = torch.nn.parallel.DistributedDataParallel(model,
                                                      device_ids=[arg.local_rank],
                                                      output_device=arg.local_rank)
</code></pre>

username: 742617000027
cooked: <p>I might have made a mistake when copying the code from the script to post here <img src="https://discuss.pytorch.org/images/emoji/apple/sweat_smile.png?v=9" title=":sweat_smile:" class="emoji" alt=":sweat_smile:"> In the code that produces the behavior mentioned in the OP, the model is wrapped actually exactly like you propsed, with the addition of the <code>find_unused_parameters</code> argument:</p>
<pre><code class="lang-auto">model = torch.nn.parallel.DistributedDataParallel(model, 
                                                  device_ids=[args.local_rank],
                                                  output_device=args.local_rank,
                                                  find_unused_parameters=True)
</code></pre>
<p>(Edited the previous post accordingly.)</p>

username: mrshenli
cooked: <p>Thanks for the clarification.</p>
<p>That’s weird. I don’t know why DDP (or sth else) would then try to access a different GPU. Does it work if you set the <code>CUDA_VISIBLE_DEVICES</code> variable to prevent the process from seeing another GPU?</p>