How can I decrease the pytorch confidence to hold too much reserved memory

username: shwgao
cooked: <p>In my case, the Pytorch seems ambitious to hold so much memory, and not free it give it back to cuda, I think it’s unreasonable, and I know it’s possible to free it through insert torch API in specific places in our code. But I want to find if there is some other ways to change the way that Pytorch allocate and cache the memory? Or, if it’s possible we can add our own memory management logic to the Pytorch backend?<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/b/0/b029b487d4f358d823af3e596428b58cf82a15d1.png" data-download-href="https://discuss.pytorch.org/uploads/default/b029b487d4f358d823af3e596428b58cf82a15d1" title="image"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b029b487d4f358d823af3e596428b58cf82a15d1_2_690x250.png" alt="image" data-base62-sha1="p8po2B86U54OQYKD6qqFuVEU5O1" width="690" height="250" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b029b487d4f358d823af3e596428b58cf82a15d1_2_690x250.png, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b029b487d4f358d823af3e596428b58cf82a15d1_2_1035x375.png 1.5x, https://discuss.pytorch.org/uploads/default/optimized/3X/b/0/b029b487d4f358d823af3e596428b58cf82a15d1_2_1380x500.png 2x" data-dominant-color="F8F5F6"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">2052×744 81 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shwgao" data-post="1" data-topic="215108">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shwgao/48/74962_2.png" class="avatar"> shwgao:</div>
<blockquote>
<p>In my case, the Pytorch seems ambitious to hold so much memory, and not free it give it back to cuda, I think it’s unreasonable,</p>
</blockquote>
</aside>
<p>It’s not unreasonable to use a caching mechanism since releasing the memory to the system via <code>cudaFree</code> (and the need to reallocate the memory again via <code>cudaMalloc</code>) using the default memory allocator is synchronizing and expensive. If you don’t want to use the caching mechanism, you can disable it via <code>PYTORCH_NO_CUDA_MEMORY_CACHING=1</code>.<br>
To implement your own allocator you can use the <code>CUDAPluggableAllocator</code> as described <a href="https://pytorch.org/docs/stable/notes/cuda.html#using-custom-memory-allocators-for-cuda">here</a> or a custom <a href="https://pytorch.org/docs/stable/generated/torch.cuda.MemPool.html#mempool"><code>MemPool</code></a>.</p>

username: shwgao
cooked: <p>Thanks, ptrblck. I have another question. If I only have 20 GB of GPU capacity, will my application run out of memory? I understand that keeping some cache can improve performance. So, can Pytorch set a threshold to keep the reserved memory below that threshold instead of not using memory caching?</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="shwgao" data-post="3" data-topic="215108">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shwgao/48/74962_2.png" class="avatar"> shwgao:</div>
<blockquote>
<p>If I only have 20 GB of GPU capacity, will my application run out of memory?</p>
</blockquote>
</aside>
<p>It’s unclear what exactly lands in the cache and if some libraries use e.g. a larger workspace since your current device has enough memory. If you are seeing a large difference between the cached and allocated memory, you could try to use <code>expandable_segments</code> as descried <a href="https://pytorch.org/docs/stable/notes/cuda.html#optimizing-memory-usage-with-pytorch-cuda-alloc-conf">here</a>.</p>
<aside class="quote no-group" data-username="shwgao" data-post="3" data-topic="215108">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shwgao/48/74962_2.png" class="avatar"> shwgao:</div>
<blockquote>
<p>So, can Pytorch set a threshold to keep the reserved memory below that threshold instead of not using memory caching?</p>
</blockquote>
</aside>
<p>Yes, you can limit the available memory in the current PyTorch process via <a href="https://pytorch.org/docs/stable/generated/torch.cuda.set_per_process_memory_fraction.html"><code>torch.cuda.set_per_process_memory_fraction</code></a>.</p>