DDP: Imbalanced load and deadlock

username: Glory_Chen
cooked: <p>Hi all, I have encountered a few strange problems when I get started to use DDP in PyTorch.</p>
<h3>Exp. setting</h3>
<p>I used 2 NVIDIA GPUs in ONE machine as well as 14 threads on 4 CPUs.<br>
I spawned 2 processes each for a GPU in a program.<br>
Each program just allocates 1+G GMEM.</p>
<h3>Questions for help</h3>
<ol>
<li>
<p>Though I have noticed to use map_location to load state_dict and so on, I found it is good for GMEM to be even across the GPUs and the somewhat strange thing was there were occasionally large fluctuations in GPU utilization, like one is 90+%, the other is ~20%, or the reverse reported. I thought DDP could lead to a more balanced load… Maybe it was because the CPUs were still in need so that the GPUs were far from busy, right?</p>
</li>
<li>
<p>When I was trying to utilize the CPUs, I ran 3 programs on these 2 GPUs. However, it was found to get stuck at a time (in the middle process of training). It appeared to be all 3 programs reduced one of their spawned processes and only left the other on the same GPU. The CPUs were not in use but the GPU utilization shown by nvidia-smi was 100% though it was obvious that the processes were stuck. I am wondering was it the case of a deadlock in that 2 programs were the best practice in the case? (might be somewhat general)</p>
</li>
</ol>
<p>(suspect the stuck is caused by?</p>
<pre><code class="lang-auto">val_loss, val_acc = self._run_epoch(...)

dist.barrier()

val_acc = torch.tensor(val_acc).unsqueeze(0).cuda()
out_val_acc = [torch.zeros_like(val_acc) for _ in range(dist.get_world_size())]
dist.all_gather(out_val_acc, val_acc)
val_acc = torch.cat(out_val_acc).mean().item()  # based on the exact division
</code></pre>
<p>Thanks &amp; Regards</p>

username: Glory_Chen
cooked: <p>A screenshot illustration for Q1.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18c3a758e179215402d4cbec1591848ba08cee56.png" alt="image" data-base62-sha1="3x4E0Xcju8kt68qyBnggjbzSQDQ" width="627" height="137"></p>

username: pritamdamania87
cooked: <aside class="quote no-group" data-username="Glory_Chen" data-post="1" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glory_chen/48/35125_2.png" class="avatar"> Glory_Chen:</div>
<blockquote>
<ul>
<li>Though I have noticed to use map_location to load state_dict and so on, I found it is good for GMEM to be even across the GPUs and the somewhat strange thing was there were occasionally large fluctuations in GPU utilization, like one is 90+%, the other is ~20%, or the reverse reported. I thought DDP could lead to a more balanced load… Maybe it was because the CPUs were still in need so that the GPUs were far from busy, right?</li>
</ul>
</blockquote>
</aside>
<p>What are the CPUs used for in your program? It could be possible that if CPUs were busy doing something and not pushing compute to GPUs, the GPU utilization might drop.</p>
<aside class="quote no-group" data-username="Glory_Chen" data-post="1" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glory_chen/48/35125_2.png" class="avatar"> Glory_Chen:</div>
<blockquote>
<ol>
<li>When I was trying to utilize the CPUs, I ran 3 programs on these 2 GPUs. However, it was found to get stuck at a time (in the middle process of training). It appeared to be all 3 programs reduced one of their spawned processes and only left the other on the same GPU. The CPUs were not in use but the GPU utilization shown by nvidia-smi was 100% though it was obvious that the processes were stuck. I am wondering was it the case of a deadlock in that 2 programs were the best practice in the case? (might be somewhat general)</li>
</ol>
</blockquote>
</aside>
<p>I’m not sure I followed this completely, could you elaborate? What are these 3 programs running? Are you running DDP in 3 processes but across 2 GPUs? In general if GPUs are stuck at 100% util it probably indicates an issue where a NCCL collective op is stuck.</p>

username: Glory_Chen
cooked: <ol>
<li></li>
</ol>
<aside class="quote no-group quote-modified" data-username="pritamdamania87" data-post="5" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<aside class="quote no-group" data-username="Glory_Chen" data-post="1" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glory_chen/48/35125_2.png" class="avatar"> Glory_Chen:</div>
<blockquote>
<ul>
<li>Though I have noticed to use map_location to load state_dict and so on, I found it is good for GMEM to be even across the GPUs and the somewhat strange thing was there were occasionally large fluctuations in GPU utilization, like one is 90+%, the other is ~20%, or the reverse reported. I thought DDP could lead to a more balanced load… Maybe it was because the CPUs were still in need so that the GPUs were far from busy, right?</li>
</ul>
</blockquote>
</aside>
<p>What are the CPUs used for in your program? It could be possible that if CPUs were busy doing something and not pushing compute to GPUs, the GPU utilization might drop.</p>
</blockquote>
</aside>
<p>I used taskset command to specify the 14 threads on the CPUs to use.</p>
<ol start="2">
<li></li>
</ol>
<aside class="quote no-group quote-modified" data-username="pritamdamania87" data-post="5" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/pritamdamania87/48/28891_2.png" class="avatar"> pritamdamania87:</div>
<blockquote>
<aside class="quote no-group" data-username="Glory_Chen" data-post="1" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glory_chen/48/35125_2.png" class="avatar"> Glory_Chen:</div>
<blockquote>
<ol>
<li>When I was trying to utilize the CPUs, I ran 3 programs on these 2 GPUs. However, it was found to get stuck at a time (in the middle process of training). It appeared to be all 3 programs reduced one of their spawned processes and only left the other on the same GPU. The CPUs were not in use but the GPU utilization shown by nvidia-smi was 100% though it was obvious that the processes were stuck. I am wondering was it the case of a deadlock in that 2 programs were the best practice in the case? (might be somewhat general)</li>
</ol>
</blockquote>
</aside>
<p>I’m not sure I followed this completely, could you elaborate? What are these 3 programs running? Are you running DDP in 3 processes but across 2 GPUs? In general if GPUs are stuck at 100% util it probably indicates an issue where a NCCL collective op is stuck.</p>
</blockquote>
</aside>
<p>Every program uses mp.spawn to spawn 2 processes on the 2 GPUs. I ran 3 parallel programs. The command can be thought of,</p>
<pre><code class="lang-auto">CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12355
CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12356
CUDA_VISIBLE_DEVICES=3,5 taskset -c 21-27,35-41 python xxx.py --world_size 2 --port 12357
</code></pre>
<p>Thanks.</p>

username: Glory_Chen
cooked: <p>Collapse with each other. Do not know why one of the 2 processes of the 2 programs run (30960, 29415) both disappear? It happens every time I run 2 parallel snippets…</p>
<pre><code class="lang-auto">mp.spawn(launch, args=(args.world_size, args), nprocs=args.world_size, join=True)
</code></pre>
<p><img src="https://discuss.pytorch.org/uploads/default/original/3X/9/7/971d17449676b3a4e0b1db82ab9de1e6587599dd.png" alt="image" data-base62-sha1="lyOqlDjxgPGlH0sSiJi6qguXEYJ" width="628" height="431"></p>

username: Glory_Chen
cooked: <p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/0/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1_2_690x107.png" alt="image" data-base62-sha1="tMr1k0oMmNSMHiOLSfqcWAj4vOF" width="690" height="107" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1_2_690x107.png, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1_2_1035x160.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/d/0/d0b8b1f3b56a9193b97d1b5fc1751f5c2f4942d1_2_1380x214.png 2x" data-dominant-color="465455"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1460×228 97.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: Glory_Chen
cooked: <p>For the second problem, maybe it is because of the reason said in <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" class="inline-onebox" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 2.1.1+cu121 documentation</a>.</p>
<blockquote>
<p>DDP processes can be placed on the same machine or across machines, but GPU devices cannot be shared across processes.</p>
</blockquote>
<p>and <a href="https://pytorch.org/docs/stable/distributed.html?highlight=init_process_group#torch.distributed.init_process_group" class="inline-onebox" rel="noopener nofollow ugc">Distributed communication package - torch.distributed — PyTorch 2.1 documentation</a>,</p>
<blockquote>
<p>If using multiple processes per machine with <code>nccl</code> backend, each process must have exclusive access to every GPU it uses, as sharing GPUs between processes can result in deadlocks.</p>
</blockquote>

username: Glory_Chen
cooked: <p>I am just reporting what I figured out for who is encountering the similar problem to refer to. In my case, it is the inconsistency caused by <strong>lr_scheduler</strong> that one process exits normally (in the logic of the program) leaving the other hanging up to wait. So it just seems to get stuck… However, I would still like to use just one NCCL backbend in a GPU as a suggested practice.</p>

username: pritamdamania87
cooked: <aside class="quote no-group quote-modified" data-username="Glory_Chen" data-post="9" data-topic="113089">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/glory_chen/48/35125_2.png" class="avatar"> Glory_Chen:</div>
<blockquote>
<p>For the second problem, maybe it is because of the reason said in <a href="https://pytorch.org/tutorials/intermediate/ddp_tutorial.html" rel="noopener nofollow ugc">Getting Started with Distributed Data Parallel — PyTorch Tutorials 1.7.1 documentation</a>.</p>
<blockquote>
<p>DDP processes can be placed on the same machine or across machines, but GPU devices cannot be shared across processes.</p>
</blockquote>
</blockquote>
</aside>
<p>Yes, the deadlock is most likely due to this. You should only use a single process per GPU.</p>

username: Glory_Chen
cooked: <p>Thanks bro. And if you have any insights or suggestions on how to balance/improve the GPU util to ~80-90% each stably, I would be appreciated.<img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: pritamdamania87
cooked: <p>Apart from running DDP are the GPUs doing any other processing? If not, what sort of processing are the 14 CPU threads doing? Are the CPU threads responsible for loading data onto the GPU?</p>
<p>Tools like nvidia’s visual profiler might help in getting a better understanding of what exactly is happening on the GPU: <a href="https://docs.nvidia.com/cuda/profiler-users-guide/index.html#mps-nvvp-import" class="inline-onebox" rel="noopener nofollow ugc">Profiler :: CUDA Toolkit Documentation</a>.</p>

username: Glory_Chen
cooked: <p>No, they do not. I think the CPUs were mainly working on image I/O loading and pre-processing (including data augmentation). But I found even if CPUs were not fully occupying all of the 14 threads (some are not 100% though htop), it often happened that one GPU util (either) is ~90%, the other is ~20% and reversed in a few second. I will take a look at the visual profiler. Thanks for the recommendation.</p>