Difference between allocated and reserved memory

pmeier: <p>I imagined that that the difference between allocated and reserved memory is the following:</p>
<ul>
<li>
<strong>allocated</strong> memory is the amount memory that is actually used by PyTorch.</li>
<li>
<strong>reserved</strong> is the allocated memory plus pre-cached memory.</li>
</ul>
<p>If that is correct the following should hold:</p>
<ul>
<li>
<strong>reserved</strong> memory <code>&gt;=</code> <strong>allocated</strong> memory</li>
<li>
<strong>reserved</strong> memory <code>~=</code> <strong>allocated</strong> memory after calling <code>torch.cuda.empty_cache()</code>
</li>
</ul>
<p>Is my understanding correct?</p>
<hr>
<p>I’m asking this since I have trouble determining the peak memory requirement for a piece of code. I’ve asked about something similar <a href="https://discuss.pytorch.org/t/determine-peak-memory-requirement/79637">before</a> (Cc <a class="mention" href="/u/ptrblck">@ptrblck</a> ).</p>
<p>My setup is as follows:</p>
<pre><code class="lang-python">for param in params:
    torch.cuda.empty_cache()
    torch.cuda.reset_peak_memory_stats()

    try:
        test_function(param)
    except RuntimeError:
        break
    finally:
        print(torch.cuda.memory_summary())
</code></pre>
<p>Here is the compressed output:</p>
<div class="md-table">
<table>
<thead>
<tr>
<th><code>param</code></th>
<th>failing</th>
<th>Allocated memory (Peak Usage)</th>
<th>GPU reserved memory (Peak Usage)</th>
</tr>
</thead>
<tbody>
<tr>
<td><code>1</code></td>
<td><code>False</code></td>
<td>15006 MB</td>
<td>16726 MB</td>
</tr>
<tr>
<td><code>2</code></td>
<td><code>False</code></td>
<td>17402 MB</td>
<td>19354 MB</td>
</tr>
<tr>
<td><code>3</code></td>
<td><code>False</code></td>
<td>19961 MB</td>
<td>22184 MB</td>
</tr>
<tr>
<td><code>4</code></td>
<td><code>True</code></td>
<td>20609 MB</td>
<td>22454 MB</td>
</tr>
</tbody>
</table>
</div><p>(Note that for <code>param==4</code> the memory report was generated after the error was raised and thus does not reflect the actual memory usage for the whole of <code>test_function</code>)</p>
<p>The memory requirement is growing approx. quadratic. A quick extrapolation for the failing <code>param</code> (<code>param==4</code>) gives 22683 MB of allocated memory and 25210 MB of reserved memory. I have 24189 MB available and no other processes are running. Thus, if my understanding about allocated and reserved memory is correct, the case should not fail.</p>
<p>Can someone explain why this is not the case?</p>

Jean_Da_Rolt: <p>Hi <a class="mention" href="/u/pmeier">@pmeier</a>, I had a similar understanding, however I just found out that the statement:</p>
<ul>
<li>
<strong>reserved</strong> memory <code>~=</code> <strong>allocated</strong> memory after calling <code>torch.cuda.empty_cache()</code>
</li>
</ul>
<p>did not hold true. If that is the case, it would be nice to have a more in depth description of the memory management here:<br>
</p><aside class="onebox allowlistedgeneric">
  <header class="source">
      <a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management" target="_blank" rel="noopener nofollow ugc">pytorch.org</a>
  </header>
  <article class="onebox-body">
    <img src="" class="thumbnail" width="" height="">

<h3><a href="https://pytorch.org/docs/stable/notes/cuda.html#cuda-memory-management" target="_blank" rel="noopener nofollow ugc">CUDA semantics — PyTorch 1.7.0 documentation</a></h3>



  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>


Darius-H: <p>So if we call torch.cuda.empty_cache(), those pre-cache memory can be used by other GPU applications. Then why <strong>reserved</strong> memory <code>!=</code> <strong>allocated</strong> memory? I’m still confused.</p>

ptrblck: <p><code>reserved</code> memory contains the <code>allocated</code> and cached memory.</p>

ndvbd: <p>What is “cached memory”? Where is the formal definition?</p>

ptrblck: <p>Memory stored in a cache, which can be reused without a reallocation.</p>

harsanyidani: <p>I understand this. But what else does it contain? I experienced multiple times that there is a significant difference between reserved and allocated memory after calling <code>torch.cuda.empty_cache()</code>. My undertanding is that they shoud be almost the same after the cache is freed. <a class="mention" href="/u/ptrblck">@ptrblck</a></p>