CUDA Out of Memory even though the model and input fit into memory

username: satyajitghana
cooked: <p>there’s this weird thing happening with me, i have a custom Residual UNet, that has about 34M params, and 133MB, and input is of batch size 512, (6, 192, 192), everything should fit into memory, although it doesn’t, it crashes consuming the entire gpu memory</p>
<p>here’s the model: <a href="https://gist.github.com/satyajitghana/b24bfb66040e7dbecc0a7d4cf7d5fa32" rel="nofollow noopener">https://gist.github.com/satyajitghana/b24bfb66040e7dbecc0a7d4cf7d5fa32</a></p>
<p>here’s the colab file running the model: <a href="https://colab.research.google.com/drive/1uOOciei9ZJ0aFyVaRlr8SYAylGPZS3Qt?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1uOOciei9ZJ0aFyVaRlr8SYAylGPZS3Qt?usp=sharing</a></p>
<p>i have no clue what the problem is, at one point of time everything had worked, on a P100, but that magic moment never came, i couldn’t figure out why it worked or why is it not working now anymore.</p>
<p>are there any memory leaks somewhere, somehow ? that i’m blind enough not to notice ?</p>
<p>even crazier thing is, sometimes even though i delete the model, i am not able to free the model’s occupied memory from gpu, it just stays there, until i reset the runtime, so the one time that the model had accidentally run successfully, i couldnt run the model in eval mode, it just crashed allocating memory.</p>

username: Isaac_Kargar
cooked: <p>you need to reduce your batch size.</p>

username: satyajitghana
cooked: <p>i tried to reduce the batch_size to 64 and ran the model, the model runs on train mode, but then i cannot run test on it, it crashes, <a href="https://colab.research.google.com/drive/1uOOciei9ZJ0aFyVaRlr8SYAylGPZS3Qt?usp=sharing" rel="nofollow noopener">https://colab.research.google.com/drive/1uOOciei9ZJ0aFyVaRlr8SYAylGPZS3Qt?usp=sharing</a><br>
i tried deleting the input tensor and clearing cache, didn’t help</p>

username: satyajitghana
cooked: <p>i was just testing which batch size would work, for now i see that batch size 32 works, but sometimes it does sometimes it doesn’t. will do a little more testing and come back here.</p>
<p>there is no training loop, i am just checking if i can atleast do a forward pass across my entire dataset.</p>

username: albanD
cooked: <p>HI,</p>
<p>You can check the doc about how we manage the CUDA memory <a href="https://pytorch.org/docs/stable/notes/cuda.html#memory-management">here</a>.<br>
In particular, this will explain why the memory is not returned to the OS when you delete your model.</p>
<p>For trying batch sizes, there are many things that can change the way the memory is allocated on the GPU and so, because of the caching allocator, will slightly change the memory usage. This is unfortunately expected and so you want to keep a liitle bit of extra memory free to make sure you don’t have any issue.</p>
<blockquote>
<p>everything should fit into memory, although it doesn’t, it crashes consuming the entire gpu memory</p>
</blockquote>
<p>I don’t think you took into account the memory used for the gradients for each parameter. And the memory used by the different states needed to compute the backward pass.<br>
These states are expected to be the most consuming in terms of memory. If you have big memory limitation, you can use modules like the checkpoint one <a href="https://pytorch.org/docs/stable/checkpoint.html">here</a> to reduce the memory used by the states at the cost of doing extra computations.</p>

username: alx
cooked: <p>reduce batch size, reduce image size, clear cache with:</p>
<pre><code class="lang-auto">  import torch, gc
  gc.collect()
  torch.cuda.empty_cache()
</code></pre>

username: satyajitghana
cooked: <p>i realized my model was way too dense to keep track of gradients, so i converted my torch.cat ops to addition ops<br>
<code>torch.cuda.empty_cache()</code> works, but the problem is it takes up training time, so i called it once after every epoch, and now everything works.</p>
<p>thanks all</p>

username: albanD
cooked: <p>I wouldn’t recommend empty_cache at all. It will be called if you’re about to run out of memory automatically, and, as you saw, it will slow down your training.</p>

username: alx
cooked: <p>What would come the closest to emptying out whats on the GPU?</p>

username: albanD
cooked: <p>The only thing pytorch puts on the GPU is the cuda runtime (that we don’t control and can’t deallocate) and Tensors.<br>
To remove the Tensors, you simply need to stop referencing them from python.</p>