It seems that pytorch.utils.checkpoint doesn't save GPU memory?

raining_day513: <p>I need to show that some technique called gradient checkpointing can really save GPU memory usage during backward propagation. When I see the result using pytorch_memlab there are two columns on the left showing <code>active_bytes</code> and <code>reserved_bytes</code>. In my testing, while active bytes read <code>3.83G</code>, the reserved bytes read <code>9.35G</code>. So why does PyTorch still reserve that much GPU memory?</p>

ptrblck: <p>The reserved memory would refer to the cache, which PyTorch can reuse for new allocations.<br>
Based on the stats you are seeing it seems that some peak memory usage might have been larger, but PyTorch is able to release it and push it back to the cache, so that it can reuse it the next time it needs memory without allocating new device memory via <code>cudaMalloc</code>.</p>

raining_day513: <p>So if I continue the reasoning with my example: Say I have a <code>4G</code> GPU card, I still cannot train the model even though the active memory usage is <code>3.83G</code>, since those cached memories up to <code>9.35G</code>  are needed for that <code>3.83G</code>?</p>

ptrblck: <p>I’m not familiar with your use case and don’t know how you were using the checkpointing util. or which operations are using the cache etc.<br>
Generally, you are trading compute for memory via checkpointing as the intermediate activations will not be stored, but recomputed.</p>

raining_day513: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="160097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>Generally, you are trading compute for memory via checkpointing as the intermediate activations will not be stored, but recomputed.</p>
</blockquote>
</aside>
<p>Yes, but I cannot just say that to people… I need to graph it so people will know “Well, now I’m convinced that PyTorch really did that.” So now my approach is to try to use this <code>pytorch_memlab</code> package.</p>
<aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="160097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>I’m not familiar with your use case and don’t know how you were using the checkpointing util. or which operations are using the cache etc.</p>
</blockquote>
</aside>
<p>OK, cool. So based on this some of my operations might cause that cached being used?</p>

ptrblck: <p>Sorry, I don’t understand your current concerns or issues, as the only thing you are describing is that checkpointing apparently doesn’t work as you are seeing memory in the internal cache without giving any details.</p>
<aside class="quote no-group" data-username="raining_day513" data-post="5" data-topic="160097">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/raining_day513/48/52553_2.png" class="avatar"> raining_day513:</div>
<blockquote>
<p>So based on this some of my operations might cause that cached being used?</p>
</blockquote>
</aside>
<p>Every memory allocation will land in the cache when the reference to the tensor is deleted to avoid the synchronizing and expensive <code>cudaMalloc</code> calls.</p>

raining_day513: <blockquote>
<p>Sorry, I don’t understand your current concerns or issues</p>
</blockquote>
<p>My concern: <a href="https://github.com/pytorch/pytorch/issues/7617#issuecomment-1223653578" class="inline-onebox" rel="noopener nofollow ugc">checkpoint(function, *args) should have the same requires_grad as function(*args) · Issue #7617 · pytorch/pytorch · GitHub</a></p>
<p>Anyway, after I re-read your first comment on this thread, it helps.</p>