How to interpret the memory information

weizuo: <p>I got the following OOM error, but I cannot understand the message precisely.</p>
<blockquote>
<p>OOM: Ran out of memory with exception: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 2; 39.59 GiB total capacity; 31.01 GiB already allocated; 21.19 MiB free; 31.83 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>From the message, seems PT reserved only 31.83GB, given the GPU capacity is 39.59GB, it should have ~8 GB free, but it only has 21.19MB free. Does it mean there are 8GB memory not managed by PT allocator? Could you help me understand where the 8GB memory goes?</p>
<p>Is the reserved memory the total memory that is allocated and cached by the memory allocator?</p>
<p>Also, I got the following memory stat print out, seems align with the error message.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a66bb23f025bac7ce02cf02cc5c716aa63e018b4.png" data-download-href="https://discuss.pytorch.org/uploads/default/a66bb23f025bac7ce02cf02cc5c716aa63e018b4" title="image"><img src="https://discuss.pytorch.org/uploads/default/original/3X/a/6/a66bb23f025bac7ce02cf02cc5c716aa63e018b4.png" alt="image" data-base62-sha1="nKdZTscyxIEfRIKZItPoigWUuk4" width="554" height="500" data-dominant-color="191919"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">image</span><span class="informations">681×614 16.1 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ptrblck: <aside class="quote no-group" data-username="weizuo" data-post="1" data-topic="132600">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weizuo/48/42419_2.png" class="avatar"> weizuo:</div>
<blockquote>
<p>Could you help me understand where the 8GB memory goes?</p>
</blockquote>
</aside>
<p>Other processes might use it, which should be visible via <code>nvidia-smi</code>. Memory fragmentation could also play a role, but I doubt you are losing so much memory due to this.</p>

weizuo: <p>I checked all memory is free via nvidia-smi when the system is idle before + after my test, so I dont think other processes are using it.</p>
<p>is it possible that some memory is allocated in the process, but not managed by the PT allocator (e.g., directly goes into the cuda allocator?)</p>

weizuo: <p>If it is fragmentation, wouldn’t that be counted in the “reserved memory” ? Only 31.83 GBs are reserved…</p>

ptrblck: <aside class="quote no-group" data-username="weizuo" data-post="4" data-topic="132600">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/weizuo/48/42419_2.png" class="avatar"> weizuo:</div>
<blockquote>
<p>is it possible that some memory is allocated in the process, but not managed by the PT allocator (e.g., directly goes into the cuda allocator?)</p>
</blockquote>
</aside>
<p>Yes, this would be possible if you are using a custom CUDA extension and are allocating memory directly there.</p>
<p>Could you try to post a minimal, executable code snippet which would reproduce this behavior on a 40GB GPU?</p>