CUDA out of memory but the numbers don't add up

username: markl
cooked: <p>I’m trying to train two nets simultaneously on my machine. When starting the second training script, I got this strange error</p>
<p><code>RuntimeError: CUDA out of memory. Tried to allocate 136.00 MiB (GPU 0; 12.00 GiB total capacity; 4.51 GiB already allocated; 130.87 MiB free; 164.77 MiB cached)</code></p>
<p>If I have 12.00 GiB total, and 4.51 GiB allocated, I should have 7.49 GiB free. Then why does allocating merely 136 MiB exhaust the memory? 136 MiB &lt; 7.49 GiB</p>
<p>Also the size of my model checkpoint is less thank 200 MB. What’s a good way to figure out why I’m using up this memory?</p>

username: ptrblck
cooked: <p>The error message claims only <code>130.87 MiB</code> are free.<br>
What is <code>nvidia-smi</code> showing?</p>
<p>Besides the parameters of the model, a large portion of the memory usage might be due to the forward activations, gradients etc.</p>

username: markl
cooked: <p>On my computer (Windows 10) nvidia-smi does not work correctly (<a href="https://stackoverflow.com/questions/44227767/nvidia-smi-does-not-display-memory-usage/44228331" class="inline-onebox" rel="noopener nofollow ugc">machine learning - nvidia-smi does not display memory usage - Stack Overflow</a>)</p>
<p>However I can use the task manager to see that the first python process is taking about 7 Gigs.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b83a4b5f11eb9ebe2205450f0bb1862e7b3823e.png" data-download-href="https://discuss.pytorch.org/uploads/default/7b83a4b5f11eb9ebe2205450f0bb1862e7b3823e" title="gpu_usage"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/b/7b83a4b5f11eb9ebe2205450f0bb1862e7b3823e.png" alt="gpu_usage" data-base62-sha1="hCELXwZHiqv0aurb83CWM1oLexw" width="690" height="156" data-dominant-color="EFF1F0"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu_usage</span><span class="informations">1096×249 19 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>After printing <code>torch.cuda.memory_allocated(device)/1e9</code> after every <code>.to(device)</code> call in my script, I’m seeing that everything remains under 0.4 until I call my <code>net.forward()</code> at which time I get the error, so I think you are right.</p>
<p>This would imply about a 20x overhead on top of just the weights + net input to reach the 7 gigs reported by task manager. This is surprising that the overhead would be that large.</p>
<p>But back to my original question, I had a thought on why the numbers do not add up. When saying <code>already allocated</code>, the error must only be reporting only the amount allocated by the current process, not the amount of GPU memory allocated period.</p>

username: tancl
cooked: <p>Hi,</p>
<p>correct me if I am wrong. I think the large amount of GPU memory is used up by the training data, the network parameters are those comparisonly small to the data loaded to GPU memory for training.</p>
<p>you might try to reduce the batch size by half and see whether this situation improve?</p>
<p>Thanks.</p>
<p>rgds,<br>
CL</p>

username: markl
cooked: <p>I don’t think it’s the training data, because I  <code>torch.cuda.memory_allocated(device)</code> after calling <code>.to(device)</code> on the input batch.</p>

username: ptrblck
cooked: <p>I’m not familiar with the Windows task manager output, but it will most likely also show the cached memory (which is allocated by PyTorch and can be reused).</p>
<p>The overhead depends also on the layer type.<br>
E.g. convolution layers have a small memory footprint regarding their parameters, but a larger memory usage regarding the output volumes.</p>
<p>This conv layer will just use 9 FP32 parameters, which is tiny:</p>
<pre><code class="lang-python">conv = nn.Conv2d(1, 1, 3, 1, 1m bias=False).cuda()
</code></pre>
<p>However, if we feed an input of <code>[10, 1, 224, 224]</code> to this layer, the output will have the same shape as the input and thus we double the memory usage.<br>
Usually the number of output channels might be even larger, so depending on your model architecture, the overhead is expected.</p>