DDP - Batch Norm Issue

soulslicer: <p>I am having the issue that everyone else has, where a model that uses BatchNorm has poorer accuracy when using DDP:</p>
<p>According to this, I am suppose to patch Batch Norm somehow:<br>
</p><aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/Microsoft/human-pose-estimation.pytorch/issues/8" target="_blank" rel="nofollow noopener">github.com/Microsoft/human-pose-estimation.pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/Microsoft/human-pose-estimation.pytorch/issues/8" target="_blank" rel="nofollow noopener">Why do you disable cudnn for batch_norm?</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2018-08-25" data-time="15:30:48" data-timezone="UTC">03:30PM - 25 Aug 18 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2018-11-06" data-time="10:59:42" data-timezone="UTC">10:59AM - 06 Nov 18 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/jin-s13" target="_blank" rel="nofollow noopener">
          <img alt="jin-s13" src="https://avatars0.githubusercontent.com/u/11788150?v=4" class="onebox-avatar-inline" width="20" height="20">
          jin-s13
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">Thank you for releasing the code. The README reads that it is required to disable cudnn for batch_norm. Would you please...</p>
</div>

<div class="labels">
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>

<pre><code class="lang-auto">def monkey_patch_bn():
    # print(inspect.getsource(torch.nn.functional.batch_norm))
    def batch_norm(input, running_mean, running_var, weight=None, bias=None,
                   training=False, momentum=0.1, eps=1e-5):
        if training:
            size = input.size()
            size_prods = size[0]
            for i in range(len(size) - 2):
                size_prods *= size[i + 2]
            if size_prods == 1:
                raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))

        return torch.batch_norm(
            input, weight, bias, running_mean, running_var,
            training, momentum, eps, False
        )
    torch.nn.functional.batch_norm = batch_norm
</code></pre>
<p>But I am not sure how to do it if my code is like this:</p>
<pre><code class="lang-auto">def convbn(in_planes, out_planes, kernel_size, stride, pad, dilation, bn_running_avg=False):
    return nn.Sequential(nn.Conv2d(in_planes, out_planes,
                                   kernel_size=kernel_size, stride=stride,
                                   padding=dilation if dilation &gt; 1 else pad,
                                   dilation=dilation, bias=False),
                         nn.BatchNorm2d(out_planes,
                                        track_running_stats=bn_running_avg))
</code></pre>

rvarm1: <p>Hi,</p>
<p>Could you try disabling the CuDNN backend with:<br>
<code>torch.backends.cudnn.enabled = False</code>? According to posts such as <a href="https://discuss.pytorch.org/t/training-performance-degrades-with-distributeddataparallel/47152/13" class="inline-onebox">Training performance degrades with DistributedDataParallel</a>, can improve training.</p>
<p>Also, have you given <code>SyncBatchNorm</code> (<a href="https://pytorch.org/docs/stable/nn.html#syncbatchnorm" rel="nofollow noopener">https://pytorch.org/docs/stable/nn.html#syncbatchnorm</a>) a try? This will make batch statistics be computed across all GPUs in usage, instead of being computed separately for the batches passed to  each device. (Note that as per the documentation, you’ll have to change your code to spawn a single process per-GPU if you’re not training that way already)</p>

soulslicer: <p>Okay but this is only useful if i am having running_mean_stats enabled right?</p>