How does "reserved in total by PyTorch" work?

abhinavdhere: <p>I got</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 10.76 GiB total capacity; 9.76 GiB already allocated; 21.12 MiB free; 9.88 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>I know that my GPU has a total memory of at least 10.76 GB, yet PyTorch is only reserving 9.88 GB.<br>
In another instance, I got a <code>CUDA out of memory error</code> in middle of an epoch and it said “<code>5.50 GiB reserved in total by PyTorch</code>”<br>
Why is it not making use of the available memory? How do I change this amount of memory reserved?</p>
<p>Using PyTorch 1.4.0 and cuda 10.1</p>

anantguptadbl: <p>You should look for the following</p>
<ul>
<li>
<p><code>torch.cuda.empty_cache()</code> This should free up the memory</p>
</li>
<li>
<p>If the memory still does not get freed up, there is a active variable in your session that is locking up memory. Restart your session and run your code again</p>
</li>
</ul>

abhinavdhere: <p><a class="mention" href="/u/anantguptadbl">@anantguptadbl</a><br>
I could manage to free up enough memory to get my network running in this instance. However, the point of this post is to try to understand this line, “reserved in total by PyTorch” and look for a way to control that. As you may have noted, I have also quoted another instance where it only reserved half of the available memory</p>

Umais: <p>I have the exact same question. I can not seem to locate any documentation on how pytorch reserves memory and the general information regarding memory allocation seems pretty scant.</p>
<p>I’m also experiencing Cuda Out of Memory errors when only half my GPU memory is being utilised (“reserved”).</p>

LuckyIceland: <p>I also got this problem. My GPU has 8GB in total yet CUDA reserves only 5G while running the program.</p>

david-macleod: <p>I am also facing this issue, 23.65 GB on the card but only 12.53GB reserved by PyTorch. The card is set to exclusive mode.</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 10.84 GiB (GPU 0; 23.65 GiB total capacity; 1.03 GiB already allocated; 10.26 GiB free; 12.53 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>It would be really helpful to understand why this memory is not being utilised</p>

Dhanush_S_R: <p>I am using Google Colab and using Hugging Face library. The code ran once but I tried to restart the session and rerun it and resulted in the following error.</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 30.00 MiB (GPU 0; 7.43 GiB
total capacity; 6.58 GiB already allocated; 30.94 MiB free; 6.79 GiB reserved in 
total by PyTorch)
</code></pre>
<p>It would be really great to understand what is causing this issue.</p>

ptrblck: <p>Based on the error message it seems the session restart might not have cleared the GPU memory.<br>
Did you change anything else or did you just restart the notebook?<br>
Could you restart the environment and check the used GPU memory?</p>

david-macleod: <p><a class="mention" href="/u/luckyiceland">@LuckyIceland</a> <a class="mention" href="/u/umais">@Umais</a> <a class="mention" href="/u/abhinavdhere">@abhinavdhere</a> see this issue which gives more details<br>
<aside class="onebox githubissue">
  <header class="source">
      <a href="https://github.com/pytorch/pytorch/issues/35901" target="_blank" rel="nofollow noopener">github.com/pytorch/pytorch</a>
  </header>
  <article class="onebox-body">
    <div class="github-row">
  <div class="github-icon-container" title="Issue">
	  <svg width="60" height="60" class="github-icon" viewbox="0 0 14 16" aria-hidden="true"><path d="M7 2.3c3.14 0 5.7 2.56 5.7 5.7s-2.56 5.7-5.7 5.7A5.71 5.71 0 0 1 1.3 8c0-3.14 2.56-5.7 5.7-5.7zM7 1C3.14 1 0 4.14 0 8s3.14 7 7 7 7-3.14 7-7-3.14-7-7-7zm1 3H6v5h2V4zm0 6H6v2h2v-2z"></path></svg>
  </div>

  <div class="github-info-container">
    <h4>
      <a href="https://github.com/pytorch/pytorch/issues/35901" target="_blank" rel="nofollow noopener">OOM error where ~50% of the GPU RAM cannot be utilised/reserved</a>
    </h4>

    <div class="github-info">
      <div class="date">
        opened <span class="discourse-local-date" data-format="ll" data-date="2020-04-02" data-time="21:16:32" data-timezone="UTC">09:16PM - 02 Apr 20 UTC</span>
      </div>


      <div class="user">
        <a href="https://github.com/david-macleod" target="_blank" rel="nofollow noopener">
          <img alt="david-macleod" src="https://avatars2.githubusercontent.com/u/17232877?v=4" class="onebox-avatar-inline" width="20" height="20">
          david-macleod
        </a>
      </div>
    </div>
  </div>
</div>

<div class="github-row">
  <p class="github-content">🐛 Bug
I am experiencing the classic out of memory error where my GPU appears to have a lot of memory still...</p>
</div>

<div class="labels">
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">module: cuda</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">topic: memory usage</span>
    <span style="display:inline-block;margin-top:2px;background-color: #B8B8B8;padding: 2px;border-radius: 4px;color: #fff;margin-left: 3px;">triaged</span>
</div>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>
<p>It seems that “reserved in total” is memory “already allocated” to tensors + memory cached by PyTorch. When a new block of memory is requested by PyTorch, it will check if there is sufficient memory left in the pool of memory which is not currently utilized by PyTorch (i.e. total gpu memory - “reserved in total”). If it doesn’t have enough memory the allocator will try to clear the cache and return it to the GPU which will lead to a reduction in “reserved in total”, however it will only be able to clear blocks on memory in the cache of which no part is currently allocated. If <em>any</em> of the block is allocated to a tensor it won’t be able to return it to GPU. Thus you can have scenarios where the tensor allocated memory + the gpu free memory is much less than the gpu total memory, because the cache is holding some which it cannot release</p>

jon: <p>Are you sure about that? It seems like the amount I have “free”, as well as the amount reserved, but not allocated by PyTorch, should be enough. Or am I misunderstanding something?</p>
<p>0.633 + (10.72-8.86) = 2.493 &gt; 1.93</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a663511a1c46fa550176ea28abe4210515b43793.png" data-download-href="https://discuss.pytorch.org/uploads/default/a663511a1c46fa550176ea28abe4210515b43793" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a663511a1c46fa550176ea28abe4210515b43793.png" alt="image" data-base62-sha1="nJW2OogHgsujaKT08ZAB3UJxp8T" width="690" height="131" data-dominant-color="0A0A0A"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">1888×361 32.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

david-macleod: <p>This looks like the exact scenario I outlined above. The (10.72-8.86)=1.86 can on be released back to CUDA (and potentially joined with the 0.633 in a new allocation to meet your requirement) if the blocks contained within that are <strong>not</strong> already partially allocated, i.e. they have not been split and some part is currently allocated to a tensor. If that has happened (which it looks like it has otherwise you wouldn’t see an error) then the only way to recover the 1.86 is to delete the tensor(s) which are allocated to the now fragmented block. Then when you run <code>torch.cuda.empty_cache()</code> the block will be released and rejoin the main CUDA pool, showing up as “free” and you should be able to allocate a new block to hold your new 1.93 tensor.</p>

jon: <p>Hmm, thanks! It’s especially interesting trying to deal with CUDA/PyTorch/deep learning memory issues, since there is both cpu ( can be fixed via <code>gc.collect()</code> and <code>del</code> statements) and gpu (as you mentioned, <code>torch.cuda.empty_cache()</code>)</p>

Adam_Kleczewski: <p>I’m running into a similar issue. Here is the error.</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 11.17 GiB total capacity; 199.97 MiB already allocated; 8.69 MiB free; 210.00 MiB reserved in total by PyTorch)</code></p>
<p>Clearly my GPU has enough memory to allocate 20.00 MiB. The limit seems to be set by the amount “reserved in total by PyTorch”. And this amount seems absurdly low to me.</p>
<p><a class="mention" href="/u/jon">@jon</a> <a class="mention" href="/u/david-macleod">@david-macleod</a> I’d be curious to hear your thoughts on whether this scenario is consistent with your interpretation of these memory specs. If so what would you recommend to fix it. Or if anyone else has an explanation or solution.</p>

ptrblck: <p>Could you check, if other processes are using the GPU and are allocating memory, since PyTorch reports only 8.69 MiB are free.</p>

Adam_Kleczewski: <p>Thanks for responding <a class="mention" href="/u/ptrblck">@ptrblck</a>.</p>
<p>There are other processes using the GPU. I’m seeing this error on a predictor service that has a 6 workers that receive prediction requests, preprocess the data, and then call the pytorch model. The workers are spun up via multiprocessing so I assume that that means each one has its own copy of the model.</p>
<p>Then I’m guessing that after making 6 copies of the model, most of the GPU memory is consumed. Then I see this error when a particularly large example is received by one of the workers.</p>
<p>Does that make sense? If so I will try reducing the number of workers to free up some of the GPU memory.</p>

ptrblck: <p>Yes, this makes sense and indeed an unexpectedly large batch in one process might create this OOM issue. You could try to reduce the number of workers or somehow guard against these large batches and maybe process them sequentially?</p>

ghasemikasra39: <p>Is not working for me:</p>
<pre><code class="lang-auto">OOM: Ran out of memory with exception: CUDA out of memory.
Tried to allocate 5.59 GiB (GPU 0; 10.92 GiB total capacity; 4.28 GiB
already allocated; 5.14 GiB free; 5.20 GiB reserved in total by PyTorch)
</code></pre>

jiapei100: <p>Same here…</p>
<pre><code class="lang-auto">......
RuntimeError: CUDA out of memory. Tried to allocate 384.00 MiB (GPU 0; 10.76 GiB total capacity; 8.84 GiB already allocated; 117.62 MiB free; 8.90 GiB reserved in total by PyTorch)
</code></pre>
<p>What a <strong>PyTorch</strong> … Extremely sad…</p>

x_zang: <p>RuntimeError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 2.00 GiB total capacity; 1.15 GiB already allocated; 13.09 MiB free; 1.16 GiB reserved in total by PyTorch)<br>
我也遇见这个问题了，请问有解决办法吗</p>

ABD-01: <p>Is this issue still not resolved! Sad.<br>
I too am facing same problem.</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 540.00 MiB (GPU 0; 4.00 GiB total capacity; 1.94 GiB already allocated; 267.70 MiB free; 2.10 GiB reserved in total by PyTorch) If reserved memory is &gt;&gt; allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF</p>
</blockquote>
<p>Only half of memory(1.94GB) is being used, rest half (2.10GB) is not being used, Or I’d say is being wasted!!</p>