How to know the exact GPU memory requirement for a certain model?

feiyuhuahuo: <p>I was doing inference for a instance segmentation model. I found the GPU memory occupation fluctuate quite much. I use both <code>nvidia-smi</code> and the four functions to watch the memory occupation: <code>torch.cuda.memory_allocated</code>, <code>torch.cuda.max_memory_allocated</code>, <code>torch.cuda.memory_reserved</code>, <code>torch.cuda.max_memory_reserved</code>. But I have no idea about the minimum memory the model needs.<br>
If I only run the model in my GPU, then the memory usage is like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/8/5/85358a8ba85b7bb045fdbc19d47fede895f54e69.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/85358a8ba85b7bb045fdbc19d47fede895f54e69" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85358a8ba85b7bb045fdbc19d47fede895f54e69_2_690x334.jpeg" alt="image" data-base62-sha1="j0qdkDU4SdPgRJdvES5oHYPpkil" width="690" height="334" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85358a8ba85b7bb045fdbc19d47fede895f54e69_2_690x334.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/8/5/85358a8ba85b7bb045fdbc19d47fede895f54e69_2_1035x501.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/8/5/85358a8ba85b7bb045fdbc19d47fede895f54e69.jpeg 2x" data-dominant-color="424648"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1338×649 309 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
10GB memory is occupied.</p>
<p>If I run another training program first to use some memory, and then run the segmentation model’s inference, then the memory usage is like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b65174d64902db6fcd48df4cc3b434f9f723c17.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/4b65174d64902db6fcd48df4cc3b434f9f723c17" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b65174d64902db6fcd48df4cc3b434f9f723c17_2_690x306.jpeg" alt="image" data-base62-sha1="aKYoBjMdEBNHlCgDgeugkI9cBbF" width="690" height="306" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b65174d64902db6fcd48df4cc3b434f9f723c17_2_690x306.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/4/b/4b65174d64902db6fcd48df4cc3b434f9f723c17_2_1035x459.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/4/b/4b65174d64902db6fcd48df4cc3b434f9f723c17.jpeg 2x" data-dominant-color="424446"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1368×608 288 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
5GB memory is occupied for the segmentation model.</p>
<p>If I run another training program with larger batch size to use some more memory first, and then run the segmentation model’s inference, then the memory usage is like:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/5/9511d29e00198dc1774a30ec8e50170ef1143352.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9511d29e00198dc1774a30ec8e50170ef1143352" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9511d29e00198dc1774a30ec8e50170ef1143352_2_690x425.jpeg" alt="image" data-base62-sha1="lgJkleLGL2RtDpEEbvpAGAjkL8m" width="690" height="425" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/5/9511d29e00198dc1774a30ec8e50170ef1143352_2_690x425.jpeg, https://discuss.pytorch.org/uploads/default/original/3X/9/5/9511d29e00198dc1774a30ec8e50170ef1143352.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/5/9511d29e00198dc1774a30ec8e50170ef1143352.jpeg 2x" data-dominant-color="4B4E50"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">923×569 170 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>With 2437MB memory, the model can also run.<br>
So I am confused about the memory usage. Even with the <code>torch.cuda.memory...</code>  like function, I can’t know how much memory my model needs. It’s always changing. It’s hard for me to choose a suitable memory GPU to deploy my model.</p>

eqy: <p>In general this can be kind of tricky to reason about, because reserved memory might not always be fully used (e.g., reserved ahead of time to speed up future allocations) and also because allocations happen in blocks and fragmentation means that reserved memory &gt; allocations.<br>
I think the closest thing you can get to a guarantee on the required memory would be to use <code>set_per_process_memory_fraction</code>: <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html#torch.cuda.set_per_process_memory_fraction" rel="noopener nofollow ugc">torch.cuda.set_per_process_memory_fraction — PyTorch 1.9.0 documentation</a> and to reduce this amount until the model cannot run to see how much memory it needs. For example, you can just keep reducing the fraction, and use the (fraction * total gpu memory) amount to get a good idea of the requirements under normal conditions.</p>
<p>Finally, after getting this estimate, I would recommend provisioning at least 100-200MiB of headroom because the memory usage of non-model things like the PyTorch/cuBLAS/cuDNN libraries may grow over time.</p>

feiyuhuahuo: <p><a class="mention" href="/u/eqy">@eqy</a> Thank for the advice. I did some tests again with <code>torch.cuda.set_per_process_memory_fraction</code>.<br>
I run the segmentation model inference on two GPUS: a 4G memory GPU and a 8G memory GPU. And I set different fractions as you told to test.<br>
The 4G GPU:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2fa50e49cb34e5abb9bbc76e7312927866259a6.png" data-download-href="https://discuss.pytorch.org/uploads/default/f2fa50e49cb34e5abb9bbc76e7312927866259a6" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2fa50e49cb34e5abb9bbc76e7312927866259a6_2_690x210.png" alt="image" data-base62-sha1="yFtQxksbYGVZsOEFKs3DJNLaHIi" width="690" height="210" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2fa50e49cb34e5abb9bbc76e7312927866259a6_2_690x210.png, https://discuss.pytorch.org/uploads/default/optimized/3X/f/2/f2fa50e49cb34e5abb9bbc76e7312927866259a6_2_1035x315.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/f/2/f2fa50e49cb34e5abb9bbc76e7312927866259a6.png 2x" data-dominant-color="1D1B1B"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1122×343 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbdd2459dc0fa3770bfc645aad77f117692a9a6f.png" data-download-href="https://discuss.pytorch.org/uploads/default/cbdd2459dc0fa3770bfc645aad77f117692a9a6f" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cbdd2459dc0fa3770bfc645aad77f117692a9a6f_2_690x226.png" alt="image" data-base62-sha1="t5sIHGGwXYJLRX4dQNdjea2Xcdh" width="690" height="226" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cbdd2459dc0fa3770bfc645aad77f117692a9a6f_2_690x226.png, https://discuss.pytorch.org/uploads/default/optimized/3X/c/b/cbdd2459dc0fa3770bfc645aad77f117692a9a6f_2_1035x339.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/b/cbdd2459dc0fa3770bfc645aad77f117692a9a6f.png 2x" data-dominant-color="1A1818"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1142×375 108 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
The 8G GPU:<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/0/a/0afa5e7854a11c7c7b6befb8533b93f83de2c39c.png" data-download-href="https://discuss.pytorch.org/uploads/default/0afa5e7854a11c7c7b6befb8533b93f83de2c39c" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0afa5e7854a11c7c7b6befb8533b93f83de2c39c_2_690x196.png" alt="image" data-base62-sha1="1z7bE9agVFLGCJoDVDpokpV9PYw" width="690" height="196" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0afa5e7854a11c7c7b6befb8533b93f83de2c39c_2_690x196.png, https://discuss.pytorch.org/uploads/default/optimized/3X/0/a/0afa5e7854a11c7c7b6befb8533b93f83de2c39c_2_1035x294.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/0/a/0afa5e7854a11c7c7b6befb8533b93f83de2c39c.png 2x" data-dominant-color="1A1919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1187×338 147 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c325925eddf7c52d94e344d9f51927da3ca4f52.jpeg" data-download-href="https://discuss.pytorch.org/uploads/default/9c325925eddf7c52d94e344d9f51927da3ca4f52" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c325925eddf7c52d94e344d9f51927da3ca4f52_2_690x231.jpeg" alt="image" data-base62-sha1="mhMm4prxjajnZVoKqlpXMmR5qgy" width="690" height="231" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c325925eddf7c52d94e344d9f51927da3ca4f52_2_690x231.jpeg, https://discuss.pytorch.org/uploads/default/optimized/3X/9/c/9c325925eddf7c52d94e344d9f51927da3ca4f52_2_1035x346.jpeg 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/9/c/9c325925eddf7c52d94e344d9f51927da3ca4f52.jpeg 2x" data-dominant-color="1C1A1A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1075×361 121 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div><br>
Both the two GPUs encountered “cuda out of memory” when the fraction &lt;= 0.4.<br>
This is still strange. For fraction=0.4 with the 8G GPU, it’s 3.2G and the model can not run. But for fraction between 0.5 and 0.8 with the 4G GPU, which memory is lower than 3.2G, the model still can run. And seems  <code>torch.cuda.set_per_process_memory_fraction</code> can only limit the pytorch reserved memory. The  reserved memory is 3372MB for 8G GPU with fraction 0.5, but nvidia-smi still shows 4643 MB. Some memory did not return to the OS.</p>

eqy: <p>Interesting, is the script doing some kind of runtime batch size adjustment after checking the amount of GPU memory? I would be surprised if the batch size is the same across GPUs with different memory usage unless something like precision of the data types is changed.</p>
<p>Yes, I don’t think there is a user-facing controllable way (or even a PyTorch controllable way short of rebuilding the libraries) to change the remaining 1GiB+ of memory usage as this is dedicated to the library code (native kernels, cuDNN, cuBLAS, etc.) that cannot be removed.</p>

feiyuhuahuo: <p><a class="mention" href="/u/eqy">@eqy</a>   I was running a model inference and the batch size are both 1. The code on two GPUs are exactly the same. I just copied one to the other. And there’s no other program running or batch size adjustment. I found the memory usage depends on the GPU type. And I found <code>torch.cuda.empty_cache()</code> can sometimes get the minimum usage. But it sometimes doesn’t work.</p>

eqy: <p>Can you post the precise CUDA error for each case? It might be useful to see explicitly what thresholds are being checked.</p>
<p>A code snippet to reproduce this could also help.</p>

feiyuhuahuo: <p>I think I know why the memory usage is high. I used <code>cudnn.benchmark = True</code> in my code. This caused a high memory usage. For the code below:</p>
<pre><code class="lang-auto">import torch
import torch.backends.cudnn as cudnn
import pdb
from torchvision.models import resnet50

net = resnet50()
net.eval()
# cudnn.benchmark = True
net = net.cuda()

with torch.no_grad():
    for i in range(100):
        img = torch.randn(1, 3, 512, 512).cuda()
        c_pred = net(img).squeeze()
        print(torch.cuda.memory_reserved()/1024/1024)
        print(torch.cuda.max_memory_reserved()/1024/1024)
        pdb.set_trace()
</code></pre>
<p>The memory usage is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/d/4/d4704e07cdeb2dcae06b2c78dfcf66f683aee447.png" alt="image" data-base62-sha1="ujjPOkqa1APvNsSF3mTSFd4aeer" width="108" height="62"><br>
And if I uncomment the <code>cudnn.benchmark = True</code>, the memory usage is:<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/0/2/02b7a595fbb5544ae189a0ed5992914072dd49d7.png" alt="image" data-base62-sha1="o2pOMLHcrpydTFt8VrgmbVKXrx" width="105" height="66"><br>
That’s quite too much memory used.  Then I used <code>torch.cuda.empty_cache()</code>, the memory usage became normal.<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/f/d/fde60085ec79c3707f61a4714ec5aa8de70f7ad8.png" alt="image" data-base62-sha1="Ae5zMu2BXEfDCUjvmBY7RGpdUdG" width="623" height="342"></p>
<p>All the results are tested on one RTX2080.</p>

eqy: <p>I’m a bit confused by what you mean here, as <code>cudnn.benchmark</code> shouldn’t ever increase the memory <em>requirements</em> of your model. If there is more memory available, the peak usage can definitely be higher as benchmark will try algorithms that require a larger workspace (temporary global memory). However, this shouldn’t affect the memory <em>requirements</em> as decreasing the available memory will filter out the algorithms that need a larger workspace.</p>

feiyuhuahuo: <p>I just comment and uncomment <code>cudnn.benchmark=True</code> to test the memory usage. And I found they differ far. I don’t know the exact mechanism behind it.<br>
Here is an answer from someone else.</p><aside class="quote" data-post="2" data-topic="90563">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/what-does-cudnn-fastest-true-work/90563/2">What does cudnn.fastest = True work?</a> 
  </div>
  <blockquote>
    It just signals the Pytorch to use the fastest implementation available for operations such as Convolution etc. when enabled, they usually consume more memory (that is cudnn.benchmark and cudnn.fastest)
  </blockquote>
</aside>


eqy: <p>Right, but the fastest available should be constrained by the available memory. So if you decrease the allowed memory, benchmark=True should never surpass it:</p><aside class="onebox githubblob" data-onebox-src="https://github.com/pytorch/pytorch/blob/d52ebf2b1bcf96a3e24e03ea86b59c8cf4a06307/aten/src/ATen/native/cudnn/Conv_v7.cpp#L205">
  <header class="source">

      <a href="https://github.com/pytorch/pytorch/blob/d52ebf2b1bcf96a3e24e03ea86b59c8cf4a06307/aten/src/ATen/native/cudnn/Conv_v7.cpp#L205" target="_blank" rel="noopener nofollow ugc">github.com</a>
  </header>

  <article class="onebox-body">
    <h4><a href="https://github.com/pytorch/pytorch/blob/d52ebf2b1bcf96a3e24e03ea86b59c8cf4a06307/aten/src/ATen/native/cudnn/Conv_v7.cpp#L205" target="_blank" rel="noopener nofollow ugc">pytorch/pytorch/blob/d52ebf2b1bcf96a3e24e03ea86b59c8cf4a06307/aten/src/ATen/native/cudnn/Conv_v7.cpp#L205</a></h4>



  <pre class="onebox">    <code class="lang-cpp">
      <ol class="start lines" start="195" style="counter-reset: li-counter 194 ;">
          <li>size_t getMaxWorkspaceSize(</li>
          <li>    const ConvolutionArgs&amp; args,</li>
          <li>    const algo_t *algo, int n_algo)</li>
          <li>{</li>
          <li>  size_t max_ws_size = 0;</li>
          <li>  size_t max_block_size = 0;</li>
          <li>  size_t tmp_bytes = 0;  // Only used for filling pointer parameters that aren't used later</li>
          <li>
          </li>
<li>  int device;</li>
          <li>  THCudaCheck(cudaGetDevice(&amp;device));</li>
          <li class="selected">  c10::cuda::CUDACachingAllocator::cacheInfo(device, &amp;tmp_bytes, &amp;max_block_size);</li>
          <li>
          </li>
<li>  for (int i = 0; i &lt; n_algo; i++) {</li>
          <li>    cudnnStatus_t err;</li>
          <li>    size_t sz;</li>
          <li>    err = getWorkspaceSize(args, algo[i], &amp;sz);</li>
          <li>    if (CUDNN_STATUS_SUCCESS != err || sz == 0 || sz &lt; max_ws_size || sz &gt; max_block_size)</li>
          <li>      continue;</li>
          <li>    max_ws_size = sz;</li>
          <li>  }</li>
          <li>  return max_ws_size;</li>
      </ol>
    </code>
  </pre>


  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>


feiyuhuahuo: <p>Yes, I did a test, benchmark=True never surpass the allowed memory. I am just surprised about the <code>max_memory_reserved</code>. Doing inference on a 24G memory GPUwith  <code>benchmark=False</code> only needs about 1.2G memory. However, the <code>max_memory_reserved</code> and the <code>nvidia-smi</code> memory usage goes to almost 12G with <code>benchmark=True</code>. Now I decide to never use it.</p>