Using convert_sync_batchnorm let my code be deadlock

username: mercer_Alex
cooked: <p>when i use the code “net = torch.nn.SyncBatchNorm.convert_sync_batchnorm(net)” to replace BN with SyncBatchNorm, the code would be deadlock like this:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/3/5/358b80317d8478c933cee1c9d01cb23d8532eed2.png" data-download-href="https://discuss.pytorch.org/uploads/default/358b80317d8478c933cee1c9d01cb23d8532eed2" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/3/5/358b80317d8478c933cee1c9d01cb23d8532eed2.png" alt="image" data-base62-sha1="7DGbsQp6mNe9LirfyjD9LoWOEAG" width="690" height="229" data-dominant-color="525252"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">880×293 30.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>it seems to be a problem with dataloader. And the relevant code is as follows<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/0/c072e4e054593df5765baa7da8b59ebffbe0eafb.png" data-download-href="https://discuss.pytorch.org/uploads/default/c072e4e054593df5765baa7da8b59ebffbe0eafb" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c072e4e054593df5765baa7da8b59ebffbe0eafb_2_690x346.png" alt="image" data-base62-sha1="rstPGHnsS8sQaHxuasxbgka4067" width="690" height="346" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c072e4e054593df5765baa7da8b59ebffbe0eafb_2_690x346.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/0/c072e4e054593df5765baa7da8b59ebffbe0eafb_2_1035x519.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/0/c072e4e054593df5765baa7da8b59ebffbe0eafb.png 2x" data-dominant-color="313030"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1114×559 69.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d04d004987395dd45ab135fd9024c5fd100d7f0.png" data-download-href="https://discuss.pytorch.org/uploads/default/8d04d004987395dd45ab135fd9024c5fd100d7f0" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d04d004987395dd45ab135fd9024c5fd100d7f0_2_690x310.png" alt="image" data-base62-sha1="k7vDjVJaSPtJA2Ngf65HSvhbCXS" width="690" height="310" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d04d004987395dd45ab135fd9024c5fd100d7f0_2_690x310.png, https://discuss.pytorch.org/uploads/default/optimized/3X/8/d/8d04d004987395dd45ab135fd9024c5fd100d7f0_2_1035x465.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/d/8d04d004987395dd45ab135fd9024c5fd100d7f0.png 2x" data-dominant-color="2F2F2F"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1066×480 45.7 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Is there any kind person to help me？Thanks.</p>

username: mrshenli
cooked: <p>The difference between <code>BatchNorm</code> and <code>SyncBatchNorm</code> is that <code>SyncBatchNorm</code> uses <code>torch.distributed.all_reduce</code> in the backward pass.</p>
<p>Two questions:</p>
<ol>
<li>What args and env vars did you pass to <code>init_process_group</code>?</li>
<li>If you program, is there any other code that launches communication ops?</li>
</ol>