Validation hangs up when using DDP and syncbatchnorm

sunshichen: <p>I’m using DDP(one process per GPU) to training a 3D UNet. I transfered all batchnorm layer inside network to syncbatchnorm with <code>nn.SyncBatchNorm.convert_sync_batchnorm</code>.</p>
<p>When doing validation at the end of every training epoch on rank 0, it always freeze at same validation steps. I think it is because of the syncbatchnorm layer. What is the correct way to do validation when DDP model has syncbatchnorm layer? Should I do validation on all ranks?</p>
<h4><a name="code-1" class="anchor" href="#code-1"></a>Code</h4>
<pre data-code-wrap="python"><code class="lang-python">for epoch in range(epochs):
    model.train()
    train_loader.sampler.set_epoch(epoch)
    for step, (data,  target) in enumerate(train_loader):
        # ...training codes
        train(model)
    if dist.get_rank() == 0:
        # ...validation codes
        model.eval()
        validate(model)
    dist.barrier()
</code></pre>
<h4><a name="version-os-2" class="anchor" href="#version-os-2"></a>Version / os</h4>
<blockquote>
<p>torch = 1.1.0<br>
ubuntu 18.04<br>
distributed backend: nccl</p>
</blockquote>
<p>Similar question:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/pytorch/pytorch/issues/20611">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/issues/20611" target="_blank" rel="noopener nofollow ugc">github.com/pytorch/pytorch</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/20611" target="_blank" rel="noopener nofollow ugc">Freeze during validation with distributed training and model with batch normalization layers</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-05-16" data-time="21:53:49" data-timezone="UTC">09:53PM - 16 May 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-05-21" data-time="00:02:36" data-timezone="UTC">12:02AM - 21 May 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/SweetVlad" target="_blank" rel="noopener nofollow ugc">
          <img alt="SweetVlad" src="https://avatars.githubusercontent.com/u/23422467?v=4" class="onebox-avatar-inline" width="20" height="20">
          SweetVlad
        </a>
      </div>
    </div>

    <div class="labels">
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          oncall: distributed
        </span>
        <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">
          triaged
        </span>
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">## ❓ Questions and Help
Hi,
I got unexpected behavior during training with tor<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">ch.distributed.DistributedDataParallel model on multiple GPUs.

- I train my model with DistributedSampler and DataLoader from torch lib. During training everything is fine, but when I validate my model after the first epoch of training, everything gets stuck  - no crash, but GPUs are showing 100 % utilization as well as CPUs, but nothing happens.

- If I remove BatchNorm2d layers from the model, everything runs just fine.

- If I use model with BatchNorm2d layers with world size 1 (only one GPU but setup through DistributedDataParallel), everything is also fine.

I assume that different batchnorm values are not correctly synced across gpus or any idea what am I doing wrong?

Thanks.

## version / os
dist backend -"nccl", Pytorch 1.1, Cuda 10, ubuntu 16.04</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


ptrblck: <p>Could you update to the latest stable release or the nightly binary and check, if you are still facing the error? <code>1.1.0</code> is quite old by now and this issue might have been already fixed.</p>

pritamdamania87: <aside class="quote no-group" data-username="sunshichen" data-post="1" data-topic="104831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshichen/48/31815_2.png" class="avatar"> sunshichen:</div>
<blockquote>
<p>What is the correct way to do validation when DDP model has syncbatchnorm layer? Should I do validation on all ranks?</p>
</blockquote>
</aside>
<p>Yes, you probably need to do validation on all ranks since SyncBatchNorm has collectives which are expected to run on all ranks. The validation is probably getting stuck since SyncBatchNorm on rank 0 is waiting for collectives from other ranks.</p>
<p>Another option is to convert the SyncBatchNorm layer to a regular BatchNorm layer and then do the validation on a single rank.</p>

sunshichen: <p>Thanks I will try it.<br>
Actually I have another question about v1.1.0 DDP.<br>
I tried to inference the model with syncbatchnorm layer <span>( Actually, it becomes batchnorm layer after load from checkpoint )</span>. The results turned to be different between:</p>
<ol>
<li>Only turn on evaluate mode.</li>
</ol>
<pre><code class="lang-python">model.eval()
# inference...
</code></pre>
<ol start="2">
<li>Manually set <code>track_running_stats</code> of each BN layer to False after <code>model.eval()</code>.</li>
</ol>
<pre><code class="lang-python">model.eval()
set_BN_track_running_stats(model, False)
# do inference..
</code></pre>
<p>It is strange that the second one is much better than first one on early epochs. Is this also a version problem?</p>
<p>P.S. However, after more epochs training, results of two inference method are similiar but still has small differences.</p>
<p>Below is sample code of <code>set_BN_track_running_stats()</code>:</p>
<pre><code class="lang-python">def set_BN_track_running_stats(module, ifTrack=True):
    if isinstance(module, torch.nn.modules.batchnorm._BatchNorm):
        module.track_running_stats = ifTrack
    for child in module.children():
        set_BN_track_running_stats(module = child, ifTrack=ifTrack)
</code></pre>

sunshichen: <p>Yes. Validate on all GPUs works. But shouldn’t <code>model.eval()</code> aotomatically disable sync process?</p>
<p>BTW, thanks for your reply.</p>

pritamdamania87: <aside class="quote no-group" data-username="sunshichen" data-post="4" data-topic="104831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshichen/48/31815_2.png" class="avatar"> sunshichen:</div>
<blockquote>
<p>Actually I have another question about v1.1.0 DDP.</p>
</blockquote>
</aside>
<p>Are you referring to PyTorch v1.1.0 here? If so, I’d suggest upgrading to PyTorch 1.7 which is the latest version to see if this problem still persists.</p>

pritamdamania87: <aside class="quote no-group" data-username="sunshichen" data-post="5" data-topic="104831">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshichen/48/31815_2.png" class="avatar"> sunshichen:</div>
<blockquote>
<p>Yes. Validate on all GPUs works. But shouldn’t <code>model.eval()</code> aotomatically disable sync process?</p>
</blockquote>
</aside>
<p>Good point, I’ve opened an issue for this: <a href="https://github.com/pytorch/pytorch/issues/48988" class="inline-onebox" rel="noopener nofollow ugc">SyncBatchNorm should avoid sync when model.eval() is True. · Issue #48988 · pytorch/pytorch · GitHub</a></p>

sunshichen: <p>Thanks greatly for your reply. I will try v1.7.</p>

sunshichen: <p>I upgrade torch to 1.7 today. The problem is gone. I think it is a 1.1.0 problem. Thanks again for you help.</p>

sunshichen: <p>Actually. I met another problem after I upgrade to V1.7.0. The result cames to be much worse than it on 1.1. Could you help me with that?</p>
<aside class="quote quote-modified" data-post="1" data-topic="106038">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshichen/48/31815_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/after-update-pytorch-from-1-1-0-to-1-7-0-ddp-model-cannot-converge/106038">After Update pytorch from 1.1.0 to 1.7.0. DDP Model cannot converge</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I trained a 3D unet model by Pytorch 1.1.0 Distributed Data Parallel. The test result is good. But after I update Pytorch to V1.7.0. Same code different result. Anyone can help me with that? Is it a syncbatchnorm problem or what? 
Even if I finetune the pretrained model(on 1.1.0). Both losses during training and validation increased a lot using v1.7.0. 
Below is the training epoch loss of two version. [loss]
  </blockquote>
</aside>


sunshichen: <aside class="quote" data-post="10" data-topic="104831">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sunshichen/48/31815_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/validation-hangs-up-when-using-ddp-and-syncbatchnorm/104831/10">Validation hangs up when using DDP and syncbatchnorm</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    Actually. I met another problem after I upgrade to V1.7.0. The result cames to be much worse than it on 1.1. Could you help me with that?
  </blockquote>
</aside>