Misunderstanding CUDA out of memory

username: Toby
cooked: <p>Hi,</p>
<p>I have a question with <code>CUDA out of memory</code>, I already know how to solve it, I just wonder the meaning of the bug.</p>
<p>GPU: RTX 2080Ti, CUDA 10.1<br>
Pytorch version: 1.6.0+cu101<br>
Model: EfficientDet-D4</p>
<p>When I trained it with the batch size is 1, it took <code>9.5 GiB</code> GPU RAM, then I tried to increase the batch size and it returned:</p>
<pre><code class="lang-auto"># Batch_size = 2
CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 11.00 GiB total capacity; 8.32 GiB already allocated; 2.59 MiB free; 8.37 GiB reserved in total by PyTorch) 

# Batch_size = 3
CUDA out of memory. Tried to allocate 64.00 MiB (GPU 0; 11.00 GiB total capacity; 8.23 GiB already allocated; 48.59 MiB free; 8.32 GiB reserved in total by PyTorch) 

# Batch_size = 4
CUDA out of memory. Tried to allocate 192.00 MiB (GPU 0; 11.00 GiB total capacity; 8.03 GiB already allocated; 90.59 MiB free; 8.28 GiB reserved in total by PyTorch) 

# Batch_size = 5
CUDA out of memory. Tried to allocate 240.00 MiB (GPU 0; 11.00 GiB total capacity; 8.06 GiB already allocated; 38.59 MiB free; 8.33 GiB reserved in total by PyTorch) 

# Batch_size = 10
CUDA out of memory. Tried to allocate 1.41 GiB (GPU 0; 11.00 GiB total capacity; 7.19 GiB already allocated; 964.59 MiB free; 7.43 GiB reserved in total by PyTorch) 
</code></pre>
<p>I am confused about how to measure the allocated memory, why the <code>already allocated</code> memory keeps decrease if I increase the batch size, and what is the meaning of <code>reserved</code> memory in that pop-up?</p>
<p>I read the code and comment in <a href="https://github.com/pytorch/pytorch/blob/b5702e2350a6cc6eef1663907175547767522424/c10/cuda/CUDACachingAllocator.cpp" rel="noopener nofollow ugc">pytorch github</a> (line 247-272). The comment mentioned the cached memory, so what is it in my case?</p>
<p><strong>Note:</strong> I noticed the memory of the driver when I killed all processes and free all tasks, it took <code>0.4~0.5/11GB</code> in my GPU</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Toby" data-post="1" data-topic="100727">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toby/48/29648_2.png" class="avatar"> Toby:</div>
<blockquote>
<p>why the <code>already allocated</code> memory keeps decrease if I increase the batch size</p>
</blockquote>
</aside>
<p>PyTorch tries to allocate the memory for the complete tensor, so increasing the batch size would also increase (some) tensors and thus the memory blocks are also bigger. If you are now running out of memory, the failed memory block might be bigger (as seen in the “tried to allocate …” message), while the already allocated memory is smaller.</p>
<aside class="quote no-group" data-username="Toby" data-post="1" data-topic="100727">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/toby/48/29648_2.png" class="avatar"> Toby:</div>
<blockquote>
<p>and what is the meaning of <code>reserved</code> memory in that pop-up?</p>
</blockquote>
</aside>
<p>Reserved memory returns the allocated and cached memory.<br>
Cached memory is used to be able to reuse device memory without reallocating it.</p>

username: Toby
cooked: <p>I am not familiar with controlling memory or memory distribution in hardware, so I cannot discuss further in the first sentence (if you have some related documents, it will help me a lot)</p>
<pre><code>    // The sum of "allocated" + "free" + "cached" may be less than the
    // total capacity due to memory held by the driver and usage by other
    // programs.
</code></pre>
<p>Follow <a href="https://github.com/pytorch/pytorch/blob/b5702e2350a6cc6eef1663907175547767522424/c10/cuda/CUDACachingAllocator.cpp" rel="noopener nofollow ugc">Pytorch github</a> (line 257-259) that I mentioned above and with your answer, I have a new question.</p>
<p>Is it should be:</p>
<pre><code> total = allocated + free + cached + driver # driver is 0.4 GiB, I mentioned above
       = allocated + free + (reserved - allocated) + driver # follow your answer
       = free + reserved + driver
</code></pre>
<p>Let take batch size = 2 to be an example, we have:</p>
<pre><code>2.59 MiB + 8.37 GiB + 0.4 GiB = 8.7725 GiB 
</code></pre>
<p>So where is the rest of the memory? <code>2.2275 GiB</code></p>