CUDA out of memory. No solution works

msuliman: <p>I am sure that this topic might have been discussed many times, but none of the proposed solutions work for me.</p>
<p>I was running my model, and then I had to interrupt the execution to do some modifications and rerun it. However, when I do so, I get the following error message:</p>
<p>RuntimeError: CUDA out of memory. Tried to allocate 2.67 GiB (GPU 0; 23.65 GiB total capacity; 6.30 GiB already allocated; 363.19 MiB free; 6.46 GiB reserved in total by PyTorch).</p>
<p>My model is small, so I am sure this is not a problem of fitting a large model into a smaller GPU memory. In fact, it was running before I interrupted its execution</p>
<p>I have tried all of the following, but none of them are working:</p>
<p>1-  Restarting the kernal.<br>
2- using torch.cuda.empty_cache(). before/after restarting the kernal.<br>
3-  Cheking the allocated meoery by: print(torch.cuda.memory_allocated()) and getting that it is zero.</p>
<p>4 - The “nvidia-smi”  shows that 67% of the GPU memory is allocated, but doesn’t show what allocates it. In fact, when you sum the individual GPU memory usage of the processes, they don’t add up to that total memory usage. The individuals processes use 1% of the total GPU memory.</p>
<p>The only thing that will work is if I restart my PC. But even after that, if I run the model and then terminate it, the out-of-memory issue will appear again.</p>
<p>Edit (1): Below is my GPU status</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06.png" data-download-href="https://discuss.pytorch.org/uploads/default/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06" title="Screenshot 2021-12-07 at 3.05.08 pm"><img src="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06_2_690x350.png" alt="Screenshot 2021-12-07 at 3.05.08 pm" data-base62-sha1="txn7An7LA0zpYYWozo0IyerkSyy" width="690" height="350" srcset="https://discuss.pytorch.org/uploads/default/optimized/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06_2_690x350.png, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06.png 1.5x, https://discuss.pytorch.org/uploads/default/original/3X/c/f/cf04cd0ae4f4febb81310ca0a77c4fb61799bd06.png 2x" data-dominant-color="EDEEEE"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">Screenshot 2021-12-07 at 3.05.08 pm</span><span class="informations">707×359 253 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Edit (2): Found this discussion <a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775" class="inline-onebox">PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error</a> by <a class="mention" href="/u/my3bikaht">@my3bikaht</a>  and used killall python through the terminal which solved my issue.</p>
<p>But I still find it weird as I am not using multithreading ( and don’t have child processes) in my model.</p>
<p>In fact, I was terminating exactly the same code and restarting it normally before, and then suddenly this issue started to happen.</p>

my3bikaht: <p>I guess you were looking for this topic:</p><aside class="quote quote-modified" data-post="1" data-topic="13775">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/rasbt/48/60_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/pytorch-doesnt-free-gpus-memory-of-it-gets-aborted-due-to-out-of-memory-error/13775">PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error</a> 
  </div>
  <blockquote>
    I noticed that 99% of the GPU RAM is still being used after and no process is listed by nvidia-smi after PyTorch aborts with an 
RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.see you:58

One could run sudo nvidia-smi --gpu-reset -i 0 as a remedy, however,  it would be nice if PyTorch could free the memory after such a “crash” – not sure if this is only a problem in my particular case, so I wanted to ask if…
  </blockquote>
</aside>


msuliman: <p>Thanks for the reply. I was going through that topic, and, killall python solved the issue</p>
<p>But I still find it weird as I am not using multithreading (I don’t have child processes) in my model.</p>
<p>In fact, I was terminating exactly the same code and restarting it normally before, and then suddenly this issue started to happen</p>

Raymond_Cool: <p>And here it is 2024 and I’m seeing exactly this - what’s going on? Seems like torch is problematic. I’m open to trying fixes, but this just started happening out of nowhere. And none of the boilerplate stuff makes any difference.</p>