Training performance degrades with DistributedDataParallel

JimFan: <p>I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU.<br>
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add <code>SyncBN</code> from pytorch 1.1, I still observe that DP &gt; DDP+SyncBN &gt; DDP without SyncBN in test accuracy.</p>
<p>I’m aware of the difference between DP and DDP’s handling of averaging/sum: <a href="https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260/16" class="inline-onebox">Is average the correct way for the gradient in DistributedDataParallel with multi nodes?</a><br>
The LR and total batch size are the same for both DP, DDP+SyncBN, and DDP.</p>
<p>If I understand correctly, DP doesn’t do SyncBN, so DP should in theory achieve the same test accuracy as DDP (given small batch size per GPU)? If we assume larger <em>effective</em> batch size leads to better result, I should expect the following test performance ranking:</p>
<p>DDP+SyncBN &gt; DP == DDP</p>
<p>but in practice, I observe: DP &gt; DDP+SyncBN &gt; DDP</p>
<p>Because DDP+SyncBN is 30% faster than DP, I really hope to solve the training gap so that I can take advantage of DDP’s superior speed. Thanks for any help!</p>

enisberk: <p>Hi Jim,<br>
From <a href="https://pytorch.org/docs/stable/_modules/torch/nn/parallel/distributed.html" rel="nofollow noopener">docs</a><br>
<code>DistributedDataParallel</code> can be used in the following two ways:<br>
(1) Single-Process Multi-GPU<br>
(2) Multi-Process Single-GPU<br>
Second method the highly recommended way to use <code>DistributedDataParallel</code>, with<br>
multiple processes, each of which operates on a single GPU. This is<br>
currently the fastest approach to do data parallel training using PyTorch<br>
and applies to both single-node(multi-GPU) and multi-node data<br>
parallel training.</p>
<p>Which one are you using ?</p>

JimFan: <p>I’m using (2). More specifically, I have one node with 8 GPUs. I launch DDP with 8 separate processes, each one owns a single GPU.</p>

enisberk: <p>I do not have a good guess then <img src="https://discuss.pytorch.org/images/emoji/apple/confused.png?v=9" title=":confused:" class="emoji" alt=":confused:"></p>
<p>Do you have any CPU heavy pre-processing, do you report only GPU performance ?<br>
Which back-end are you using, NCCL ?</p>

JimFan: <p>By “performance”, I mean the classification accuracy. Somehow DDP+SyncBN achieves worse test accuracy than DP, so there must be some problematic differences in the numeric algorithm. The speed isn’t the issue here. Thanks!</p>

enisberk: <p>My mistake, I got it wrong. Thanks for clarification.</p>

pietern: <p>I can only comment on the differences between DP and DDP w.r.t. batch normalization. With DP your module is replicated before each call to <code>forward</code>, which means that only the BN stats from the first replica are kept around. With DDP each process keeps their own version of BN. And with SyncBN you’ll end up with stats that are “more averaged” than the stats kept when using DP, because they only include stats for a batch in a single replica, instead of all replicas.</p>

Mr.Z: <p>I also encountered this performance issue with DistributedDataParallel, hope someone could give a solution. <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"></p>

Mr.Z: <p>I found the problem in my code, it’s because of the cudnn batch norm. According to <a href="https://github.com/Microsoft/human-pose-estimation.pytorch/issues/8" rel="nofollow noopener">this github issue</a>, the solution is to edit the batchnorm part in torch/nn/functional.py or set torch.backends.cudnn.enabled = False.</p>

JingLi: <p>Could edit the batchnorm part in torch/nn/functional.py work for sync bn?<br>
<aside class="onebox githubblob">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/blob/91d28026f8e8386d45dfb7ddab84c906479dead1/torch/nn/modules/batchnorm.py#L450" target="_blank" rel="nofollow noopener">github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/91d28026f8e8386d45dfb7ddab84c906479dead1/torch/nn/modules/batchnorm.py#L450" target="_blank" rel="nofollow noopener">pytorch/pytorch/blob/91d28026f8e8386d45dfb7ddab84c906479dead1/torch/nn/modules/batchnorm.py#L450</a></h4>
<pre class="onebox"><code class="lang-py"><ol class="start lines" start="440" style="counter-reset: li-counter 439 ;">
<li>        else:  # use exponential moving average</li>
<li>            exponential_average_factor = self.momentum</li>
<li>
</li>
<li>    world_size = 1</li>
<li>    process_group = torch.distributed.group.WORLD</li>
<li>    if self.process_group:</li>
<li>        process_group = self.process_group</li>
<li>    world_size = torch.distributed.get_world_size(process_group)</li>
<li>
</li>
<li>    # fallback to framework BN when synchronization is not necessary</li>
<li class="selected">    if world_size == 1 or (not self.training and self.track_running_stats):</li>
<li>        return F.batch_norm(</li>
<li>            input, self.running_mean, self.running_var, self.weight, self.bias,</li>
<li>            self.training or not self.track_running_stats,</li>
<li>            exponential_average_factor, self.eps)</li>
<li>    else:</li>
<li>        return sync_batch_norm.apply(</li>
<li>            input, self.weight, self.bias, self.running_mean, self.running_var,</li>
<li>            self.eps, exponential_average_factor, process_group, world_size)</li>
<li>
</li>
<li>@classmethod</li>
</ol></code></pre>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
The batch norm in torch.nn.functional is used just for evaluation. I think editing this would do nothing to sync batch norm. How do you edit the file to make sync bn work normally?</p>

Mr.Z: <p>You are right, although the performance improves after disable cudnn, the gap still remains. I can’t figure out the problem and now I have to use nn.DataParallel <img src="https://discuss.pytorch.org/images/emoji/apple/slightly_frowning_face.png?v=9" title=":slightly_frowning_face:" class="emoji" alt=":slightly_frowning_face:"> .</p>

YueshangGu: <p><a class="mention" href="/u/mr.z">@Mr.Z</a> Do you find the  problem? I also get a very worse accuracy when use SyncBN + DDP for batchsize=16( 4 GPUs on one node, 4 images for each GPU), and when I use DataParallel + SyncBN, evrything is OK.</p>

Sergii_Makarevych: <p>Same here. Performance of DDP model is weaker than one trained on a single GPU. Playing with lr/bs does not help. As number of GPUs in DPP training grows - performance degrades.</p>
<p>Has anyone found the solution ?</p>
<p>UPDT: the reason was found for my case. When training DDP model we need to use <code>DistributedSampler</code> which is passed to <code>Dataloader</code>. We need to <code>train_dataloader.sampler.set_epoch(epoch)</code> on every epoch start.</p>

makslevental: <p>having the same issue (DP much better validation metrics than DDP). setting</p>
<p><code>torch.backends.cudnn.enabled = False</code></p>
<p>slows my runtime down by 3x.</p>
<p>monkey patching <code>torch.nn.functional.batch_norm</code></p>
<pre><code class="lang-python">def monkey_patch_bn():
    # print(inspect.getsource(torch.nn.functional.batch_norm))
    def batch_norm(input, running_mean, running_var, weight=None, bias=None,
                   training=False, momentum=0.1, eps=1e-5):
        if training:
            size = input.size()
            size_prods = size[0]
            for i in range(len(size) - 2):
                size_prods *= size[i + 2]
            if size_prods == 1:
                raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))

        return torch.batch_norm(
            input, weight, bias, running_mean, running_var,
            training, momentum, eps, False
        )
    torch.nn.functional.batch_norm = batch_norm
</code></pre>
<p>doesn’t seem to help.</p>
<p><code>train_dataloader.sampler.set_epoch(epoch)</code> doesn’t seem to help either.</p>
<p>EDIT:</p>
<p>what does seem to work is dividing my <code>lr</code> by my <code>world_size</code>, although i’m not sure why.</p>

Beinan_Wang: <p><a class="mention" href="/u/yueshanggu">@YueshangGu</a> Hi, How do you use  DataParallel + SyncBN at the same time? I though SyncBN only works with DistributedDataParallel.</p>

Sergii_Makarevych: <p>Another issue spotted in my case: a model has to be transferred to proper device before wrapping into DDP.</p>

farleylai: <p>Our experience with Kinetics 400 using PyTorch 1.3 on a node with two GPUs as follows:</p>
<blockquote>
<p>Single GPU &gt; DP (-0.2%) &gt; DP w/ sync BN (-0.3%)</p>
</blockquote>
<p>Single GPU serves as the baseline for DP and DP w/ sync BN.<br>
The tradeoff with distributed training is understandable but sync BN causing worse accuracy is not trivial to ignore.</p>

jinyuanfeng: <p>My setting is same with you, just testing in HMDB51. I also get the results as follows:<br>
DP&gt;DP w/sync BN. Now, Do you find the solution to deal with this issue?</p>

zhi_lim: <p>Hi guys, where can i find the code of SyncBN?</p>

ginobilinie: <p>Maybe the learning rate is the problem?</p>