Torch.distributed.broadcast deadlock

cindybrain: <p>when I train a model using DDP in 4 GPUs and evaluate it  in one GPU with args.local_rank==0, I want to broadcast the top1 to other GPUs. but I got the deadlock. The GPUs (local_rank=1,2,3) just enter the next command without blocking to get the broadcast results.</p>
<p>The code is shown below. It was execuated after finishing training one epoch.</p>
<pre><code class="lang-auto">if args.local_rank == 0:
      top1, top5 = test(net, testloader, criterion, False)
torch.distributed.broadcast(torch.tensor(top1).cuda(args.local_rank),src=0, async_op=False)
print("local rank:{}, top1:{}".format(args.local_rank, top1))
</code></pre>
<p>The result is shown below. The process was hanged after print the following information:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7503c6306b1a8c7dfb139e7d9ac314f0769697c1.png" data-download-href="https://discuss.pytorch.org/uploads/default/7503c6306b1a8c7dfb139e7d9ac314f0769697c1" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/5/7503c6306b1a8c7dfb139e7d9ac314f0769697c1.png" alt="image" data-base62-sha1="gH9WUu170wh0ZB2ffusNd3VYIJr" width="690" height="201" data-dominant-color="ECE6D3"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1122×328 22.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there anyone who met the same problem?</p>

huahuanZ: <p>Can you try this way</p>
<pre><code class="lang-auto">if args.local_rank == 0:
    top1, top5 = test(net, testloader, criterion, False)
    top1 = torch.tensor(top1).cuda(args.local_rank)
else:
    top1 = torch.tensor(0.).cuda(args.local_rank)
torch.distributed.broadcast(top1,src=0, async_op=False)
print("local rank:{}, top1:{}".format(args.local_rank, top1.item()))
</code></pre>

cindybrain: <aside class="quote no-group" data-username="cindybrain" data-post="1" data-topic="123280">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/9fc348/48.png" class="avatar"> cindybrain:</div>
<blockquote>
<pre><code class="lang-auto">if args.local_rank == 0:
      top1, top5 = test(net, testloader, criterion, False)
torch.distributed.broadcast(torch.tensor(top1).cuda(args.local_rank),src=0, async_op=False)
print("local rank:{}, top1:{}".format(args.local_rank, top1))
</code></pre>
</blockquote>
</aside>
<p>Thanks for your reply! But it still doesn’t work<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/1/911942453b0eaa4adb820274bd633249042b4880.png" alt="image" data-base62-sha1="kHBlHKKrCOQzjnBd5VDhez1iH72" width="602" height="153"></p>

mrshenli: <p>Hey <a class="mention" href="/u/cindybrain">@cindybrain</a>, that’s weird. Could you please share a self-contained repro? Thanks!</p>

cindybrain: <p>Hi, <a class="mention" href="/u/mrshenli">@mrshenli</a><br>
I created a mini-repo: <a href="https://github.com/SHu0421/Question-Repo" class="inline-onebox" rel="noopener nofollow ugc">GitHub - SHu0421/Question-Repo</a>. you can directly run it by<br>
<code>bash train.sh</code>. my torch version is 1.8.1 and the cuda version is 10.2.<br>
I run the code on four Tesla v100 GPUs (one node). It hung out as before with all GPUs usage 100%</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de5e5cebde7e73b26bbb482b7285c37b88de2494.png" data-download-href="https://discuss.pytorch.org/uploads/default/de5e5cebde7e73b26bbb482b7285c37b88de2494" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/d/e/de5e5cebde7e73b26bbb482b7285c37b88de2494.png" alt="image" data-base62-sha1="vJaayatGDnILKi2aoUwbb3P0aGM" width="690" height="470" data-dominant-color="E7E0CF"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1324×902 64.9 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>