AMP uses more GPU memory and slows training

username: julien_p
cooked: <p>Hi,</p>
<p>I noticed that training a ResNet-50 with AMP on my new laptop with RTX3070 takes much more GPU memory than without AMP. It is not a code issue because I am able to run the same code on a workstation with an Nvidia Tesla p100 with the opposite result.</p>
<p>Laptop:<br>
Ubuntu 20.04 with Kernel 5.8 (and 5.11)<br>
GPU: RTX3070<br>
Latest Nvidia drivers installed with Ubuntu GUI software and updates (CUDA 11.2)<br>
command numba -s returns no error regarding CUDA install. Same for nvidia-smi.<br>
Training  without AMP: 3.9 GB VRAM<br>
Training with AMP: 7.4 GB VRAM<br>
GPU memory consumption is stable during training</p>
<p>I installed Pytroch with:<br>
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c nvidia<br>
(Also tried Pytorch preview 1.9)</p>
<p>Workstation:<br>
GPU: Nvidia tesla p100<br>
CUDA 10.2<br>
Training  without AMP: 3.5 GB VRAM<br>
Training with AMP: 2.5 GB VRAM</p>
<p>Do you have any idea why I have this strange behavior on my laptop?</p>
<p>Thanks in advance <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=9" title=":slight_smile:" class="emoji" alt=":slight_smile:"></p>

username: tom
cooked: <p>I would suggest to double-check your PyTorch version on the laptop (by printing <code>torch.__version__</code>) to make sure that you got a very fresh relelease/nightly build. It seems really easy to have conda trick you into using an old version.</p>
<p>As explained by <a class="mention" href="/u/ptrblck">@ptrblck</a> , here, there was a bug not too long ago:</p><aside class="quote" data-post="3" data-topic="115608">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/mixed-precision-increases-memory-in-meta-learning/115608/3">Mixed precision increases memory in meta-learning?</a> <a class="badge-category__wrapper " href="/c/mixed-precision/27"><span data-category-id="27" data-drop-close="true" class="badge-category "><span class="badge-category__name">mixed-precision</span></span></a>
  </div>
  <blockquote>
    Could you update to the latest stable release or the nightly and rerun the test? 
We hit a caching issue some time ago for linear layers (and fixed it). Based on the output you are seeing, I don’t think the problem is that AMP uses more memory in particular, but that the memory usage is clearly increasing/leaking.
  </blockquote>
</aside>

<p>Best regards</p>
<p>Thomas</p>

username: julien_p
cooked: <p>Hi <a class="mention" href="/u/tom">@tom</a>, thank you for your suggestion. Printing torch.<strong>version</strong> confirms I have 1.8.1 and 1.9.0.dev20210527 in another env.</p>
<p>Best regards</p>
<p>Julien</p>

username: tom
cooked: <p>Thank you for double-checking. The other thread I’d recommend  is</p>
<aside class="quote quote-modified" data-post="1" data-topic="120525">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/thomas0809/48/38007_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/torch-cuda-amp-cannot-speed-up-on-a100/120525">Torch.cuda.amp cannot speed up on A100</a> <a class="badge-category__wrapper " href="/c/mixed-precision/27"><span data-category-id="27" data-drop-close="true" class="badge-category "><span class="badge-category__name">mixed-precision</span></span></a>
  </div>
  <blockquote>
    I am trying to use torch.cuda.amp to speed up training. The following code works well on V100 (120s w/o amp and 67s w/ amp), but cannot get a reasonable speedup on A100 (53s w/o amp and 50s w/ amp). 
I am using the most recent NGC docker <a href="https://ngc.nvidia.com/catalog/containers/nvidia:pytorch" rel="noopener nofollow ugc">nvcr.io/nvidia/pytorch:21.04-py3</a>, which has CUDA 11.3, pytorch 1.9.0. 
The strange thing is if I change the model to resnet101, both V100 and A100 can get speedup, but not for the swin transformer on A100. See the following table for the running time (in sec) on…
  </blockquote>
</aside>

<p>again <a class="mention" href="/u/ptrblck">@ptrblck</a> 's a hero for looking at this.</p>
<p>About the memory: With AMP’s automatic casting so for very small networks, the memory cost of the tensors that are there twice (in FP32 originally and cast to FP16) might be larger than the benefit. (Assuming you use PyTorch’s memory counting, if you use nvidia-smi you’d might also run into caching allocator things.</p>
<p>Best regards</p>
<p>Thomas</p>

username: julien_p
cooked: <p>Thank you again <a class="mention" href="/u/tom">@tom</a>, I understood the memory difference between my laptop and workstation.</p>
<p>Using memory_allocated() both my laptop and workstation allocate 438 MB GPU memory to tensors with AMP, and 435 MB without AMP.<br>
But using memory_reserved() my laptop reserve 6.4 GB with AMP and 2.9 GB without AMP; the workstation reserves 1.63 GB with AMP and 2.79 GB without AMP.</p>
<p>So the AMP reduces Pytorch memory caching on Nvidia P100 (Pascal architecture) but increases memory caching on RTX 3070 mobile (Ampere architecture). I was expecting AMP to decrease memory allocation/reserved, not to increase it (or at least the same). As I saw in a thread that FP32 and FP16 tensors are not duplicated in GPU memory.</p>
<p>I also tried to train a ResNet-152 (224*224 images, batch size 32).  AMP is definitely slowing training on my laptop:</p>
<p>Laptop (minutes/epoch):<br>
FP32: 2:02, TF32: 2:03, AMP: 2:20</p>
<p>Workstation (minutes/epoch):<br>
FP32: 1:49, TF32: 1:46, AMP: 1:45</p>
<p>I was expecting TF32 and AMP to increase throughput on Ampere architecture, not to decrease it. Do you have any idea?</p>

username: ptrblck
cooked: <p>The binaries are missing cutlass kernels in the statically linked cudnn as described <a href="https://github.com/pytorch/pytorch/issues/50153">here</a>, so you would have to build it from source and check the performance again.</p>

username: julien_p
cooked: <p>Thanks a lot <a class="mention" href="/u/ptrblck">@ptrblck</a>, that solved the issue!</p>
<p>Do you know when the bug will be fixed in conda cudnn?</p>