BCE Loss 2-3 times slower on multi GPU (update: CUDAs sync takes long time)

username: keltis
cooked: <p>Hello,</p>
<p>I am aware that there are several other topics about slow computation on multi GPU - none of them seems to fit to the behaviour I am facing.</p>
<p>I have a model that utilizes Huggingface BERT and has a total of 111,450,730 params. When I set  BERTs <code>requires_grad</code> to False, the model has 2,369,386 params. Apart from the number of parameters, the problematic behaviour is the same: BCE Loss computation is up to three times as expensive when switching to Dataparallel using two instead of one GPU (also changing the batch size does not have any effect).</p>
<p>My model is pretty much standard, however <code>F.binary_cross_entropy(pred, target)</code> takes ~50 ms in single GPU mode and ~150 ms when training on two GPUs.</p>
<pre><code class="lang-auto">def update_model_for_trainer(batch):
    model.train()
    optimizer.zero_grad()
    input, target = batch
    pred = model(input)
    loss = F.binary_cross_entropy(pred, target) # 50 ms without and 150 ms with Dataparallel
    loss.backward()
    optimizer.step()
    return loss.item()
</code></pre>
<p>Things tried so far:</p>
<ul>
<li>Explicitely moving prediction and target to <code>cuda:0</code>
</li>
<li>Integrating the loss calculation into the model, after that I used <code>sum(loss).backwards()</code>; had no effect</li>
</ul>
<p>I would be really thankful for some tips what to do next to investigate the problem.</p>
<p>Thank you in advance<br>
Thomas</p>

username: ptrblck
cooked: <p>CUDA operations are asynchronous, so you have to synchronize the code before starting and stopping the timer via <code>torch.cuda.synchronize()</code>.<br>
Could you add it to your code and profile the operations again, please?</p>

username: keltis
cooked: <p>Hi!</p>
<p>Following your advice, the situation changed. Please look at the following picture, which summarizes the average computation times of the train function:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/f/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d.png" data-download-href="https://discuss.pytorch.org/uploads/default/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d" title="Screenshot 2020-03-15 at 15.06.24"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d_2_690x277.png" alt="Screenshot 2020-03-15 at 15.06.24" data-base62-sha1="mHZfJ9XJft5t9HM3iAtWL7qRi57" width="690" height="277" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d_2_690x277.png, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d_2_1035x415.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/9/f/9f28c3839dea355b5bb7429a026f9ee0cbfa6d5d_2_1380x554.png 2x" data-dominant-color="F5F5F5"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2020-03-15 at 15.06.24</span><span class="informations">2070×832 210 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>It turns out that the loss computation indeed can be ignored. Regarding the <code>model + sync</code>:</p>
<p>*) When in DP mode, on both devices <code>model.forward(...)</code> is entered and left at nearly identical ms timestamps. The difference is max 5 ms (which is ok I guess). The average time diff between entering and leaving <code>model.forward(...)</code> is constant at ~20-40 ms depending on using DP or not.</p>
<p>Is the conclusion correct that the problem is, that the cudas are out of sync?<br>
Since the batches contain data of different lengths, could this be the origin of the problem?</p>
<p>Greetings<br>
Thomas</p>
<p>edit: <a class="mention" href="/u/ptrblck">@ptrblck</a> this was supposed to be an answer to you - missed the reply button <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

username: ptrblck
cooked: <p>Thanks for the update! The profiling looks better now. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>
<aside class="quote no-group" data-username="keltis" data-post="3" data-topic="73252">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/keltis/48/21687_2.png" class="avatar"> keltis:</div>
<blockquote>
<p>Since the batches contain data of different lengths, could this be the origin of the problem?</p>
</blockquote>
</aside>
<p>That could be a possible reason.<br>
However, we generally recommend to use DistributedDataParallel with a single process per GPU to avoid the scatter and gather of the model in each iteration, which might add some overhead.</p>