DDP on 8 gpu work much worse then on single

hadaev8: <p>So i read this thread and some other, googled and etc</p><aside class="quote quote-modified" data-post="1" data-topic="47152">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jimfan/48/1296_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/training-performance-degrades-with-distributeddataparallel/47152">Training performance degrades with DistributedDataParallel</a> <a class="badge-category__wrapper " href="/c/distributed/12"><span data-category-id="12" data-drop-close="true" class="badge-category "><span class="badge-category__name">distributed</span></span></a>
  </div>
  <blockquote>
    I’m training a conv model using DataParallel (DP) and DistributedDataParallel (DDP) modes. For DDP, I only use it on a single node and each process is one GPU. 
My model has many BatchNorm2d layers. Given all other things the same, I observe that DP trains better than DDP (in classification accuracy). Even if I add SyncBN from pytorch 1.1, I still observe that DP &gt; DDP+SyncBN &gt; DDP without SyncBN in test accuracy. 
I’m aware of the difference between DP and DDP’s handling of averaging/sum: <a href="https://discuss.pytorch.org/t/is-average-the-correct-way-for-the-gradient-in-distributeddataparallel-with-multi-nodes/34260/16" class="inline-onebox">Is av…</a>
  </blockquote>
</aside>

<p>My model is seq to seq variable len input (tacotron 2).</p>
<p>This is how looks like loss<br>
<a href="https://i.imgur.com/NAm5Woe.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/NAm5Woe.png</a></p>
<p>Any suggestions? Any way how to debug it?</p>
<p>Also if i remove all batchnorm from model, it raise unused variable error, how it work?</p>

ptrblck: <p>Are you using any reference implementation as your code base or did you write the complete code yourself?<br>
Could you compare your implementation to <a href="https://github.com/NVIDIA/DeepLearningExamples/tree/master/PyTorch/SpeechSynthesis/Tacotron2">this one</a> written by NVIDIA?</p>

hadaev8: <p>Im using fork of this repo<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://github.githubassets.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/NVIDIA/tacotron2" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/1728152?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/NVIDIA/tacotron2" target="_blank" rel="nofollow noopener">NVIDIA/tacotron2</a></h3>

<p>Tacotron 2 - PyTorch implementation with faster-than-realtime inference - NVIDIA/tacotron2</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
<br>
(looks like same with repo you linked)<br>
But i have a lot of changes</p>

ptrblck: <p>In that case I would focus on the error (about the unused variable) by reverting the changes and make sure your model still trains.</p>
<p>I assume the error you mentioned claims that no parameters were found which require gradients during the backward pass.<br>
If so, it’s usually a sign, that all parameters were frozen or that the computation graph was detached at some point.</p>

hadaev8: <p>So, this problem happen then i comment out batchnorm layers.<br>
Then i return it back, it runs (still worse then single gpu).</p>

hadaev8: <p>Another observation: upgraded torch from 1.2 to 1.3.1 and now i have OOM error on same setup.</p>

ptrblck: <p>If the original code base works and you see this unwanted behavior, I would still recommend to triage the bug by removing your changed parts.<br>
The error points to a (partly) frozen model. I’m not familiar with the use case, so it might be a red herring. Anyway, it might be a good starter for debugging.</p>
<p>We are aware of the higher memory usage due to some added functionalities in a tensor method and are thinking about different approaches to fix this.</p>

hadaev8: <p>Most major difference between my model and nvidia’s is this modification.</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bfs18/tacotron2/blob/master/train.py#L251">
  <header class="source">

      <a href="https://github.com/bfs18/tacotron2/blob/master/train.py#L251" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bfs18/tacotron2/blob/master/train.py#L251" target="_blank" rel="noopener nofollow ugc">bfs18/tacotron2/blob/master/train.py#L251</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="241" style="counter-reset: li-counter 240 ;">
          <li>x, y = model.parse_batch(batch)</li>
          <li>y_pred = model(x)</li>
          <li></li>
          <li>loss = criterion(y_pred, y)</li>
          <li>if model.mi is not None:</li>
          <li>    # transpose to [b, T, dim]</li>
          <li>    decoder_outputs = y_pred[0].transpose(2, 1)</li>
          <li>    ctc_text, ctc_text_lengths, aco_lengths = x[-2], x[-1], x[4]</li>
          <li>    taco_loss = loss</li>
          <li>    mi_loss = model.mi(decoder_outputs, ctc_text, aco_lengths, ctc_text_lengths)</li>
          <li class="selected">    if hparams.use_gaf:</li>
          <li>        if i % gradient_adaptive_factor.UPDATE_GAF_EVERY_N_STEP == 0:</li>
          <li>            safe_loss = 0. * sum([x.sum() for x in model.parameters()])</li>
          <li>            gaf = gradient_adaptive_factor.calc_grad_adapt_factor(</li>
          <li>                taco_loss + safe_loss, mi_loss + safe_loss, model.parameters(), optimizer)</li>
          <li>            gaf = min(gaf, hparams.max_gaf)</li>
          <li>    else:</li>
          <li>        gaf = 1.0</li>
          <li>    loss = loss + gaf * mi_loss</li>
          <li>else:</li>
          <li>    taco_loss = loss</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Without gradient balancing lost curve much better (red)<br>
<a href="https://i.imgur.com/uONtHJj.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/uONtHJj.png</a></p>
<p>Does it matter if loss calculated in forward or outside?<br>
Also is it good idea in general to balance loss value like this?<br>
Tacotron by its own have 2 losses</p><aside class="onebox githubblob" data-onebox-src="https://github.com/bfs18/tacotron2/blob/master/loss_function.py#L19">
  <header class="source">

      <a href="https://github.com/bfs18/tacotron2/blob/master/loss_function.py#L19" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/bfs18/tacotron2/blob/master/loss_function.py#L19" target="_blank" rel="noopener nofollow ugc">bfs18/tacotron2/blob/master/loss_function.py#L19</a></h4>



    <pre class="onebox"><code class="lang-py">
      <ol class="start lines" start="9" style="counter-reset: li-counter 8 ;">
          <li>mel_target, gate_target = targets[0], targets[1]</li>
          <li>mel_target.requires_grad = False</li>
          <li>gate_target.requires_grad = False</li>
          <li>gate_target = gate_target.view(-1, 1)</li>
          <li></li>
          <li>_, mel_out, mel_out_postnet, gate_out, _ = model_output</li>
          <li>gate_out = gate_out.view(-1, 1)</li>
          <li>mel_loss = nn.MSELoss()(mel_out, mel_target) + \</li>
          <li>    nn.MSELoss()(mel_out_postnet, mel_target)</li>
          <li>gate_loss = nn.BCEWithLogitsLoss()(gate_out, gate_target)</li>
          <li class="selected">return mel_loss + gate_loss</li>
      </ol>
    </code></pre>



  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
Do you think it should be balanced too?</p>
<p>About batchnorm thing, i don’t understand how batchnorm affect params.<br>
I mean it change only existed variables, do not freeze layers or something.<br>
It work with batchnorm layers and raise error without.<br>
<a href="https://i.imgur.com/Tc3yACY.png" class="onebox" target="_blank" rel="noopener nofollow ugc">https://i.imgur.com/Tc3yACY.png</a><br>
May be because without batchnorm i have only one layer in module?<br>
Or it doesnt matter?</p>
<p>I made another thread about memory usage (it is not only distributed thing) with example, hope it helps.</p><aside class="quote" data-post="1" data-topic="63398">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/hadaev8/48/16280_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/50-vram-used-on-torch-1-3-compared-to-1-2/63398">+ 50% VRAM used on torch 1.3 compared to 1.2</a> 
  </div>
  <blockquote>
    1.3 
<a href="https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU" class="onebox" target="_blank" rel="noopener nofollow ugc">https://colab.research.google.com/drive/104LtQ1zIioIOMQEPgVve77m5Rd4Gm0wU</a> 
1.2 
<a href="https://colab.research.google.com/drive/1y4LF1a90PYfKvFgQw6fCMYKtIFK07mVr" class="onebox" target="_blank" rel="noopener nofollow ugc">https://colab.research.google.com/drive/1y4LF1a90PYfKvFgQw6fCMYKtIFK07mVr</a> 
Everything same except torch version. 
At the end nvidia-smi show 11gb and 7 gb memory used.
  </blockquote>
</aside>


hadaev8: <p>Is here any pytorch options to debug this situation?<br>
I have made another test, gap seems to be too bad.<br>
                    <a href="https://i.imgur.com/ldW0uQw.png" target="_blank" class="onebox" rel="nofollow noopener">
            <img src="https://i.imgur.com/ldW0uQw.png" width="690" height="313">
          </a>

</p>

dabs: <p>Not sure this is the case here, but in my case I was using autocast and GradScaler. I had both set to enabled=False. According to the docs this should mean these should have no effect, which was in fact the case with a single GPU and using DP.</p>
<p>However, with DDP I found that introducing these increased variance in the training and validation loss significantly, deteriorating model accuracy overall. According to the <a href="https://pytorch.org/docs/stable/notes/amp_examples.html#working-with-multiple-gpus" rel="noopener nofollow ugc">docs</a> autocast and GradScaler shouldn’t adversely affect DDP, but it did just that in my case. Not sure why, but I assume it has to do with gradient synchronization in DDP.</p>