Training with Half Precision

Amir_Rosenfeld: <p>I was wondering if anyone tried training on popular datasets (imagenet,cifar-10/100) with half precision, and with popular models (e.g, resnet variants)?</p>

colesbury: <p>It works, but you want to make sure that the BatchNormalization layers use float32 for accumulation or you will have convergence issues. You can do that by something like:</p>
<pre><code class="lang-auto">model.half()  # convert to half precision
for layer in model.modules():
  if isinstance(layer, nn.BatchNorm2d):
    layer.float()
</code></pre>
<p>Then make sure your input is in half precision.</p>
<p>Christian Sarofeen from NVIDIA ported the ImageNet training example to use FP16 here:<br>
<aside class="onebox whitelistedgeneric">
  <header class="source">
      <img src="https://assets-cdn.github.com/favicon.ico" class="site-icon" width="16" height="16">
      <a href="https://github.com/csarofeen/examples/tree/fp16_examples_cuDNN-ATen/imagenet" target="_blank" rel="nofollow noopener">GitHub</a>
  </header>
  <article class="onebox-body">
    <img src="https://avatars0.githubusercontent.com/u/22205833?s=400&amp;v=4" class="thumbnail onebox-avatar" width="60" height="60">

<h3><a href="https://github.com/csarofeen/examples/tree/fp16_examples_cuDNN-ATen/imagenet" target="_blank" rel="nofollow noopener">csarofeen/examples</a></h3>

<p>A set of examples around pytorch in Vision, Text, Reinforcement Learning, etc.</p>


  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>We’d like to clean-up the FP16 support to make it more accessible, but the above should be enough to get you started.</p>

sid: <p>This is great! Is there documentation on when/where half precision can be used? For example, it doesn’t seem like half precision computation is supported on CPU, but I only discovered this by giving it a shot.</p>

Sergey_Milyaev: <p><a class="mention" href="/u/colesbury">@colesbury</a>, could you suggest the right way for conversion of fp16 inputs to BatchNorm with fp32 parameters? I think, this modification is now mentioned in <a href="http://docs.nvidia.com/deeplearning/sdk/mixed-precision-training/index.html#training_pytorch" rel="nofollow noopener">NVIDIA documentation</a> as special batch normalization layer, but I couldn’t find any implementation example.</p>

ashtonsix: <p>Hi <a class="mention" href="/u/colesbury">@colesbury</a>,</p>
<p>I converted my batch-norm layers back to floats with this code, which works:</p>
<pre><code class="lang-auto">def batchnorm_to_fp32(module):
    if isinstance(module, nn.modules.batchnorm._BatchNorm):
        module.float()
    for child in module.children():
        batchnorm_to_fp32(child)
    return module
</code></pre>
<p>But autograd doesn’t like me using a mixture of float &amp; half tensors, running <code>loss.backward()</code> leads to an error like this:</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/d/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255.png" data-download-href="https://discuss.pytorch.org/uploads/default/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255" title="error: &quot;expected HalfTensor, recieved FloatTensor&quot;"><img src="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255_2_690x369.png" alt="error: &quot;expected HalfTensor, recieved FloatTensor&quot;" data-base62-sha1="v6oXxFTMq2zoNRl5wkzp17l3Enb" width="690" height="369" srcset="https://discuss.pytorch.org/uploads/default/optimized/2X/d/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255_2_690x369.png, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255_2_1035x553.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/2X/d/d9fcac4610a382f8b6aa5b4f0158e83c5a08a255_2_1380x738.png 2x" data-dominant-color="FAD6D9"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">error: "expected HalfTensor, recieved FloatTensor"</span><span class="informations">1422×761 115 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Do you think you can help me?</p>
<p>Thank you if yes! <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"></p>

kenfehling: <p>Is it the same situation with weight norm? It should use float32?</p>

Marat: <p>What speed gain do you achieve using FP16 instead of FP32 on pytorch (on resnet or simular)?</p>

tstandley: <p>It really depends. On a Titan X or P100, you get about 15% speedup for all the architectures I’ve tried. On a Titan V or V100, I get about a 50% speedup for resnet50 and 2x speedup on Xception, probably because of the way tensor cores work. The 2x speedup actually makes the Titan V worth it if you are going to be training a lot of networks that use grouped convolution. You also get to use double the batch size because the smaller floats all fit in vram.</p>
<p>I should say that the speed-up isn’t painless. I’ve had issues with fp16 overflow at times. Usually these are fixable, but you only find out about them after investing a significant amount of time training.</p>
<p>I also worry about the added complexity leading me to make wrong conclusions about my experimental outcomes. (i.e. could this issue be an fp16 issue? Or thinking one model works better but really the other model was faulty.)</p>

tstandley: <p>You also need to be sure to maintain a full 32 bit copy of your parameters. This helps stability substantially.</p>

mcarilli: <p>We’ve developed a lightweight, open-source set of Pytorch tools to enable easier, more numerically stable mixed precision training:  <a href="https://github.com/nvidia/apex" rel="nofollow noopener">https://github.com/nvidia/apex</a>.  Mixed precision means that the majority of the network uses FP16 arithmetic (reducing memory storage/bandwidth demands and enabling Tensor Cores for gemms and convolutions), while a small subset of operations are executed in FP32 for improved stability.</p>
<p>Highlights include:</p>
<ul>
<li>Amp, a tool that executes all numerically safe Torch functions in FP16, while automatically casting potentially unstable operations to FP32.  Amp also automatically implements dynamic loss scaling.  Amp is designed to offer maximum numerical stability, and most of the speed benefits of pure FP16 training.</li>
<li>FP16_Optimizer, an optimizer wrapper that automatically implements FP32 master weights for parameter updates, as well as static or dynamic loss scaling.  FP16_Optimizer is designed to be minimally invasive (it doesn’t change the execution of Torch operations) and offer almost all the speed of pure FP16 training with significantly improved numerical stability.</li>
<li>apex.parallel.DistributedDataParallel, a distributed module wrapper that achieves high performance by overlapping computation with communication during backward().  Apex DistributedDataParallel is useful for both pure FP32 as well as mixed precision training.</li>
</ul>
<p>Full API documentation can be found <a href="https://nvidia.github.io/apex/" rel="nofollow noopener">here</a>.</p>

mcarilli: <p>Our <a href="https://github.com/NVIDIA/apex/tree/master/examples" rel="nofollow noopener">examples page</a> demonstrates the use of FP16_Optimizer and Apex DistributedDataParallel.  Amp examples are coming soon, and Amp’s use is thoroughly discussed in its <a href="https://github.com/NVIDIA/apex/tree/master/apex/amp" rel="nofollow noopener">README</a>.</p>
<p>Give Apex a try and let us know what you think!</p>
<p>sorry for double post, the forum page told me “new users may only post 2 links at a time” or something along those lines.</p>

Adam_Dziedzic: <p>The link to csarofeen/examples does not work any more. You can find an example here: <a href="https://discuss.pytorch.org/t/fp16-on-pytorch-0-4/20984/2?u=adam_dziedzic">Fp16 on pytorch 0.4</a></p>

RB_Zhang: <p>Hi thanks for your explanation.<br>
May I ask why the BN must use float32, does that mean BN us different from other layers, like conv, linear, etc?</p>

williamFalcon: <p>I’d say the easiest way to use and not make a mistake is to use PyTorch Lightning with<br>
Trainer(use_amp=True).</p>
<p>This will train your model using 16-bit.</p>
<p><a href="https://pytorch-lightning.readthedocs.io/en/0.6.0/trainer.html" class="onebox" target="_blank" rel="nofollow noopener">https://pytorch-lightning.readthedocs.io/en/0.6.0/trainer.html</a></p>

Falak_Shah: <p>Thanks <a class="mention" href="/u/mcarilli">@mcarilli</a>. Apex was very useful to us in our project.</p>

VasudevGupta7: <p>any suggestions on using float16 with transformers. Should I keep some layers in float32 just like batch-normalization is recommended to keep in float32?</p>

ptrblck: <p>I would generally recommend to use the automatic mixed precision package (via <a href="https://pytorch.org/docs/stable/amp.html"><code>torch.cuda.amp</code></a>), which uses casts the input to the appropriate <code>dtype</code> for each method.</p>

VasudevGupta7: <p>okay thanks. Should we keep val_step under autocast scope as well for fair comparison between tr_loss &amp; val_loss?</p>

ptrblck: <p>Yes, you can also use autocasting during the validation.<br>
Especially if you plan on using it for the test dataset (or deployment) I would use it.</p>

quangduyhcmut: <p>I used torch.cuda.amp tools to training an u-net-like network but my loss function gave NaN. I guess this is overflow’s problem when using fp16. Can you give me some advice to overcome this? Thank you so much!</p>