Cuda out of memory despite consuming only 60% memory

ashwini: <p>I’m getting cuda out of memory error. The error is shown below</p>
<blockquote>
<p>RuntimeError: CUDA out of memory. Tried to allocate 256.00 MiB (GPU 0; 23.65 GiB total capacity; 21.65 GiB already allocated; 242.88 MiB free; 22.55 GiB reserved in total by PyTorch)</p>
</blockquote>
<p>I’m able to train the model after I reduce the batch_size. When I checked the output of <code>nvidia-smi</code> , I see a 40% of memory is still free. Here is the output.<br>
<div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10513e9fde75cdeb8bea768a45194344822a896f.png" data-download-href="https://discuss.pytorch.org/uploads/default/10513e9fde75cdeb8bea768a45194344822a896f" title="cudo ouf of memory"><img src="https://discuss.pytorch.org/uploads/default/original/3X/1/0/10513e9fde75cdeb8bea768a45194344822a896f.png" alt="cudo ouf of memory" data-base62-sha1="2klHJjGDBLCcdCzbcLO0WzpJ2FN" width="690" height="304" data-dominant-color="3A4041"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">cudo ouf of memory</span><span class="informations">798×352 34.5 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>What could be possible reason for this?<br>
pytorch: 1.4<br>
cuda: 10.2<br>
input_size: (512, 512, 4)<br>
using half-precision</p>
<p>More information: The plot of gpu-utilization is shown below<br>
<img src="https://discuss.pytorch.org/uploads/default/original/3X/3/0/305bb7962f4076e8b86c7ff82dbe191098c4f66d.png" alt="gpu_usage2" data-base62-sha1="6TNq8M6BkvCXRURrTlGSkWiWxIN" width="614" height="306"></p>
<p>The numbers at each peak represent the batch_size. It seems the initial memory requirement is much higher than memory needed afterward. Can someone explain?</p>

ptrblck: <p>The higher peak memory usage might e.g. indicate different behaviors.<br>
E.g. some initial copies are needed, if you are passing expanded tensors, which are needed in a contiguous format.<br>
Also, if you are running your script with <code>cudnn.benchmark</code> the initial iteration will benchmark different cudnn algorithms and select the fastest one, which might also create a memory peak (however, this should now yield an OOM issue).</p>

ashwini: <p>I’ve checked the code <code>cudnn.benchmark</code> is <code>True</code>. Though It is very unlikely that pytorch adds more than 10 gb for  8 more images.<br>
I’m unable to understand the meaning of “some initial copies are needed if you are passing expanded tensors, which are needed in a contiguous format”. Can you please explain this? It might help me in locating the problem.</p>

ptrblck: <p><a href="https://discuss.pytorch.org/t/understanding-cdist-function/76296/12">Here</a> is an example of what I had in mind.</p>

ashwini: <p>Haven’t used contiguous anywhere in the code. I’m using catalyst DL and segementation_models_pytorch. Catalyst DL uses contiguous call only in lovasz loss and I’m not using it.</p>
<p>Just checked <code>cudnn.benchmark</code> makes no difference.</p>

abhi901: <p><strong><a href="https://pytorch.org/docs/master/generated/torch.no_grad.html#torch.no_grad" rel="nofollow noopener">Try to use : torch.no_grad() &amp; model.eval() if not using.</a></strong><br>
Most important : torch.no_grad() as it impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations in validation loop or test loop.</p>