Torch.cuda.amp vs Nvidia apex?

username: braindotai
cooked: <p>I’ve been confused lately regarding torch.cuda.amp and Nvidia apex. What each of them does? (As such I know that both of them does mixed precision) How do they differ and when to use which?</p>

username: ptrblck
cooked: <p><code>torch.cuda.amp</code> just landed recently in the nightly builds and is the recommended way to use mixed-precision training.</p>
<p>CC <a class="mention" href="/u/mcarilli">@mcarilli</a>, who was working on pushing this utility into the PyTorch core. <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: mcarilli
cooked: <p>tl;dr <a href="https://pytorch.org/docs/master/amp.html" rel="nofollow noopener"><code>torch.cuda.amp</code></a> is the way to go moving forward.</p>
<p>We published Apex Amp last year as an experimental mixed precision resource because Pytorch didn’t yet support the extensibility points to move it upstream cleanly.  However, asking people to install something separate was a headache.  Extension building and forward/backward compatibility were particular pain points.</p>
<p>Given the benefits of automatic mixed precision, it belongs in Pytorch core, so moving it upstream has been my main project for the past six months.  I’m happy with <code>torch.cuda.amp</code>.  It’s more flexible and intuitive than Apex Amp, and repairs many of Apex Amp’s known flaws.  Apex Amp will shortly be deprecated (and to be honest I haven’t been working on it for a while, I focused on making sure <code>torch.cuda.amp</code> covered the most-requested feature gaps).</p>
<p>Prefer <code>torch.cuda.amp</code>, early and often.  It supports a <a href="https://pytorch.org/docs/master/notes/amp_examples.html" rel="nofollow noopener">wide range of use cases</a>.  If it doesn’t support your network for some reason, file a Pytorch issue and tag <a class="mention" href="/u/mcarilli">@mcarilli</a>.  In general, prefer native tools for versioning stability (that means <code>torch.nn.parallel.DistributedDataParallel</code> too) because they’re tested and updated as needed for each master commit or binary build.</p>
<p>Apex will remain as a source of utilities that can be helpful, e.g. fast fused optimizers, but forward+backward compatibility across all Pytorch versions can’t be guaranteed.  Don’t take a dependency on Apex unless you want to try those.</p>

username: braindotai
cooked: <p>Thanks <a class="mention" href="/u/ptrblck">@ptrblck</a> and <a class="mention" href="/u/mcarilli">@mcarilli</a>.</p>

username: Yangmin
cooked: <p>Hello. I have only single GPU and a very heavy model, so I am running out of memory…<br>
Recently, I figured out that there is mixed precision training which saves the GPU memory during training.</p>
<p>Is NVIDIA apex an older tool to use mixed training?</p>
<p>I read some post and saw your comment there is a pytorch implementation which is called torch.cuda.amp.</p>
<p>How do I use it and what is nightly builds you commented above?</p>

username: seungjun
cooked: <p>Hi <a class="mention" href="/u/yangmin">@Yangmin</a><br>
You may want to refer to:<br>
<a href="https://pytorch.org/docs/stable/notes/amp_examples.html" class="onebox" target="_blank" rel="noopener nofollow ugc">https://pytorch.org/docs/stable/notes/amp_examples.html</a></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Yangmin" data-post="5" data-topic="74994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yangmin/48/43502_2.png" class="avatar"> Yangmin:</div>
<blockquote>
<p>Is NVIDIA apex an older tool to use mixed training?</p>
</blockquote>
</aside>
<p>Yes, <code>apex.amp</code> was our first implementation of mixed-precision training, is deprecated now, and replaced with <code>torch.cuda.amp</code>. <a class="mention" href="/u/seungjun">@seungjun</a> posted the examples above to see its usage.</p>
<aside class="quote no-group" data-username="Yangmin" data-post="5" data-topic="74994">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/yangmin/48/43502_2.png" class="avatar"> Yangmin:</div>
<blockquote>
<p>How do I use it and what is nightly builds you commented above?</p>
</blockquote>
</aside>
<p>You don’t need to install a nightly release anymore, as <code>torch.cuda.amp</code> is available in the stable releases already.</p>

username: zzibc
cooked: <p>Hi mcarilli!<br>
It is feasible to combine <code>torch.cuda.amp</code> and <a href="https://pytorch.org/docs/stable/ddp_comm_hooks.html#torch.distributed.algorithms.ddp_comm_hooks.default_hooks.fp16_compress_hook" rel="noopener nofollow ugc">fp16_compress_hook</a>? Casting gradients to <code>float16</code> and allreducing those <code>float16</code> gradient tensors are helpful to reduce communication cost.</p>

username: mcarilli
cooked: <p>Yes, the action of those hooks should be composable with the things amp does.</p>