Slower Mixed precision than fp32 on 2080 Ti RTX

Ushk: <p>I apologize if I’ve missed something obvious here - this question relates to issues I am having timing mixed precision vs float32 computation.</p>
<p>I have two servers - one with Pytorch 1.5 and Cuda 10.1, and the other with Pytorch 1.6 and Cuda 11.0. As far as I know there are no Pytorch CUDA 11.0 binaries, so that pytorch was compiled with 10.1. Both have 2080 Ti RTX GPU cards.</p>
<p>In both servers, I time pure fp32 computation as being significantly faster than mixed precision and I can’t work out why.</p>
<p>I am aware that for the Pytorch1.6 server the mismatch between CUDA versions is not ideal, but I’m still not sure why there should be an issue on the Pytorch 1.5 server.</p>
<p>My times are as follows:<br>
Pytorch 1.5:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/b/d/bd9894b72f0dd1f4aa36f8c37510edb55e0c3d46.png" alt="image" data-base62-sha1="r3f8Zuocnix3jLkf6Tlv0XARf1k" width="347" height="71"></p>
<p>Pytorch 1.6<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/9/3/93d98ad6ae9850881ed3f3fbfa8d800c6c8eb4c7.png" alt="image" data-base62-sha1="l5WgLFG5oEzVRRWV60Yo2Hhcnvp" width="392" height="95"></p>
<p>I’ve attached gists for the two scripts that I’m using to compute times below.</p>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642">
  <header class="source">

      <a href="https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642</a></h4>

  <h5>gistfile1.txt</h5>
  <pre><code class="Text">import random
import os
import numpy as np
import time
import torch
import torch.nn as nn
from apex import amp

def set_seed(seed: int):
    """Set all seeds to make results reproducible (deterministic mode).</code></pre>
   This file has been truncated. <a href="https://gist.github.com/Ushk/96be97cd595ee63ae5cd6cebe415e642" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<aside class="onebox githubgist" data-onebox-src="https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14">
  <header class="source">

      <a href="https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14" target="_blank" rel="noopener nofollow ugc">gist.github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14" target="_blank" rel="noopener nofollow ugc">https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14</a></h4>

  <h5>gistfile1.txt</h5>
  <pre><code class="Text">import os
import time
import torch
import torch.nn as nn
import numpy as np
import random
from apex import amp
from torch.cuda.amp import autocast

def set_seed(seed: int):</code></pre>
   This file has been truncated. <a href="https://gist.github.com/Ushk/8831348c071ddea6a90c7dddad124d14" target="_blank" rel="noopener nofollow ugc">show original</a>

<p>
</p>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>Torch versions:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9f0690730f6da76b655d72994cbf10610be99bf.png" data-download-href="https://discuss.pytorch.org/uploads/default/a9f0690730f6da76b655d72994cbf10610be99bf" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/9/a9f0690730f6da76b655d72994cbf10610be99bf.png" alt="image" data-base62-sha1="oflLGw52beQU5meIoajx7NBlRtt" width="690" height="172" data-dominant-color="38162D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">719×180 20.2 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18017d4d24160f2dc5d080b996e3731e1a2215c5.png" data-download-href="https://discuss.pytorch.org/uploads/default/18017d4d24160f2dc5d080b996e3731e1a2215c5" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/8/18017d4d24160f2dc5d080b996e3731e1a2215c5.png" alt="image" data-base62-sha1="3qmEkibLUUftThkQ05ZoKQ84sAZ" width="690" height="168" data-dominant-color="38152D"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">730×178 20.3 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Output of nvcc --version<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/3/f30ebe0759061af1a014b914f18526a9901d0515.png" alt="image" data-base62-sha1="yGbBQXfzXEyV0exAXdoDGU3cZKJ" width="494" height="89"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/7/77f7a703d728a730ddae0d7c67aade36c65bdc97.png" alt="image" data-base62-sha1="h7hpwZtnjS4MNL2IicZNj3HovhJ" width="495" height="89"></p>
<p>Nvidia-smi:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5e62f66546fadb9cc11cd7cb9b21df560fc6460.png" data-download-href="https://discuss.pytorch.org/uploads/default/a5e62f66546fadb9cc11cd7cb9b21df560fc6460" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/5/a5e62f66546fadb9cc11cd7cb9b21df560fc6460.png" alt="image" data-base62-sha1="nFBX1ESMAKP8gTH3BviKStGEjsc" width="690" height="303" data-dominant-color="3C1A32"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">805×354 35.8 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24bf90b86a63f63cdf61e8550537de2dc8e371bc.png" data-download-href="https://discuss.pytorch.org/uploads/default/24bf90b86a63f63cdf61e8550537de2dc8e371bc" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/2/4/24bf90b86a63f63cdf61e8550537de2dc8e371bc.png" alt="image" data-base62-sha1="5f5CaRkwlY81baFEYKdocdmsZZy" width="690" height="336" data-dominant-color="3A1930"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">809×394 36.4 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Based on this comment/thread:</p><aside class="onebox githubissue" data-onebox-src="https://github.com/facebookresearch/fairseq/issues/464#issuecomment-462064590">
  <header class="source">

      <a href="https://github.com/facebookresearch/fairseq/issues/464#issuecomment-462064590" target="_blank" rel="noopener nofollow ugc">github.com/facebookresearch/fairseq</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewBox="0 0 14 16" aria-hidden="true"><path fill-rule="evenodd" d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/facebookresearch/fairseq/issues/464#issuecomment-462064590" target="_blank" rel="noopener nofollow ugc">Huge Training time for FP16 on Turing/2080 GPU</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2019-01-22" data-time="14:16:07" data-timezone="UTC">02:16PM - 22 Jan 19 UTC</span>
      </div>

        <div class="date">
          closed <span class="discourse-local-date" data-format="ll" data-date="2019-02-19" data-time="06:29:12" data-timezone="UTC">06:29AM - 19 Feb 19 UTC</span>
        </div>

      <div class="user">
        <a href="https://github.com/kalyangvs" target="_blank" rel="noopener nofollow ugc">
          <img alt="kalyangvs" src="https://avatars.githubusercontent.com/u/42022935?v=4" class="onebox-avatar-inline" width="20" height="20">
          kalyangvs
        </a>
      </div>
    </div>

    <div class="labels">
    </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">I am training en-fr on GeForce RTX 2*2080 ti with nvidia driver version 410.79 (<span class="show-more-container"><a href="" rel="noopener" class="show-more">…</a></span><span class="excerpt hidden">although the latest is 410.93) with CUDA 10 and fp16 flag enabled
With the following command 

python3.6 -m torch.distributed.launch --nproc_per_node 2 train.py data-bin --arch transformer --share-all-embeddings --optimizer adam --adam-betas '(0.9, 0.98)' --clip-norm 0.0 --lr-scheduler inverse_sqrt --warmup-init-lr 1e-07 --warmup-updates 4000 --lr 0.0007 --min-lr 1e-09 --dropout 0.1 --weight-decay 0.0 --criterion label_smoothed_cross_entropy --label-smoothing 0.1 --max-tokens 4096 --update-freq 8 --save-dir /experiments/transformer_base_fr_en --ddp-backend=no_c10d --fp16

But a part of my training log is 
| epoch 001:   0%| | 73/21742 [09:49&lt;49:52:09,  8.29s/it, loss=14.730, nll_loss=14.568, ppl=24291.78, wps=4262, ups=0.1, wpb=60960, bsz=1607, num_updates=69, lr=1.21733e-05, gnorm=3.811, clip=0%, oom=0, loss_scale=8.000, wall=987, train_wall=586]

which says 50 more hours are left, but when fp16 is disabled it takes around 10 hours per epoch on the whole 36 M sentences. Why is there such a huge drop ?
Any flags to add ? Or might be a mismatch between driver version and graphic card ?

Please help..</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>

<p>I would expect there to be a speedup on a 2080 Ti  - is this correct?</p>

seungjun: <p>I experienced a similar phenomenon when running very small networks with <code>amp.autocast</code>. In that case, pure fp16 inference was the fastest followed by fp32 and <code>amp.autocast</code> was the slowest.<br>
Autocasting requires type conversion of input and the corresponding layers before the main computation. Type casting could take more time than the saved time from main computation in lower precision. In larger networks, the gains in computation will be more than the loss from the type conversion.</p>

ptrblck: <p>In your scripts you are rightfully synchronizing before starting the timer, but no synchronization is used when you are stopping the timer.<br>
Thus your timing might be wrong and in fact you might profile the PyTorch overhead and the kernel launch times.</p>
<p>Add syncs before starting and stopping the timer and execute the real workload (forward/backward) inside a loop to get more stable results.</p>
<p>Also, setting <code>torch.backends.cudnn.benchmark = True</code> and <code>cudnn.deterministic = False</code> might also help, as cudnn will profile different algos and will select the fastest one.<br>
Note that this setup would increase the first iteration time (for forward and backward) significantly, as the profiling would be executed, so you should add warmup iterations.</p>
<p>I would also recommend to use the CUDA10.2 binaries, which ship with cudnn7.6.5.32 or the nightly CUDA11.0 binaries, which come now with cudnn8.0.4.30.</p>

Ushk: <p>Thanks for the heads up <a class="mention" href="/u/ptrblck">@ptrblck</a>!</p>
<p>Changed the two bools as suggested, added an extra synch call before stopping the timer, add an additional backward pass before timing for profiling for fp32 and a fwd/backward autocast pass. New timing with the additional synch call for the Pytorch 1.5 server:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74ef6cb8803230f49eba6714ae2e288bc17eeb45.png" alt="image" data-base62-sha1="gGslNzmyOZyvXFz0UDbk0glQUUB" width="336" height="68"></p>
<p>So the timings went down - due to changing the bool values, as suggested - but still the wrong (intuitively) order.</p>
<p><a class="mention" href="/u/seungjun">@seungjun</a> - thanks as well for the tip. I tried with Resnet18 and Resnet50 and the timings are even, which seems more reasonable, though is still a bit surprising.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/2/9/2902719ee0645c553febc36d270b76416bd3d82f.png" alt="image" data-base62-sha1="5QMOlZ7DkGqHPlFp5LfiCzlWJEH" width="362" height="70"><br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/6/f62f634100e734798bbca7ffddc5de32349b8d55.png" alt="image" data-base62-sha1="z7QZ605TjwMqKO1w11MlR3GtAvH" width="335" height="50"></p>
<p>The 1.6 server is currently occupied, will be able to update tomorrow. Unfortunately, I’m using a cloud provider that only has certain docker images - in this case:</p>
<ul>
<li>pytorch:1.5.0-cuda10.1-cudnn7-devel</li>
<li>pytorch:1.6.0-cuda10.1-cudnn7-devel<br>
and the two <a class="mention" href="/u/ptrblck">@ptrblck</a> mentioned aren’t available :(.</li>
</ul>

ptrblck: <p>The CUDA10.2 binaries are available <a href="https://pytorch.org/get-started/locally/">here</a> and you would only have to select this particular version.<br>
The CUDA11 nightly binaries can be installed by using <code>cudatoolkit=11.0</code> in the conda install command.</p>