PyTorch doesn't free GPU's memory of it gets aborted due to out-of-memory error

rasbt: <p>I noticed that 99% of the GPU RAM is still being used after and no process is listed by nvidia-smi after PyTorch aborts with an</p>
<pre><code class="lang-auto">RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1518244421288/work/torch/lib/THC/generic/THCStorage.see you:58
</code></pre>
<p>One could run <code>sudo nvidia-smi --gpu-reset -i 0</code> as a remedy, however,  it would be nice if PyTorch could free the memory after such a “crash” – not sure if this is only a problem in my particular case, so I wanted to ask if others have the same issue (before I post a GitHub issue)</p>
<p>EDIT 1: I should mention that the memory is freed if I e.g., interrupt or kill a process, it’s just the <code>RuntimeError: cuda runtime error (2) : out of memory</code> which doesn’t seem to free the memory</p>
<p>EDIT 2: It does not seem to be just a reporting issue by nvidia-smi as I can’t run even smaller networks on that card after I encountered that issue</p>

bitcurian: <p>That’s odd. I have been dealing with out of memory issues but the memory always cleans up after the crash.</p>
<p>Pytorch: 0.3.1<br>
Cuda:9.1<br>
Perhaps you could list your environmental setup.</p>

rasbt: <p>I am currently using PyTorch 0.3.1 with Cuda 9.0 via Miniconda. The graphics cards where this happens are Tesla K80’s (tried multiple different ones), and I have had the same issue with Tesla K20’s.</p>

bitcurian: <p>I honestly don’t think it’s cause of the GPU. I’ve tried it on a GTX 970m and a Titan XP and it works fine.<br>
The only other difference from your setup is that I installed using pip on a virtual environment.</p>

rasbt: <p>Hm, that’s interesting! There are 8 Tesla K80’s in that rack, and I just checked, the two ones where I “crashed” the PyTorch code (via <code>RuntimeError: cuda runtime error (2) : out of memory</code>) still occupy that memory (GPU 1 &amp; 2, GPU 7 is the one I am currently using)</p>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/2X/3/30a93f7c7c14ee9afda6679aab16ebbd85b453ac.png" data-download-href="https://discuss.pytorch.org/uploads/default/30a93f7c7c14ee9afda6679aab16ebbd85b453ac" title="14%20PM"><img src="https://discuss.pytorch.org/uploads/default/original/2X/3/30a93f7c7c14ee9afda6679aab16ebbd85b453ac.png" alt="14%20PM" data-base62-sha1="6WtwU8Wron5ct5ss6CKQ8xG0wYA" width="434" height="500" data-dominant-color="332F33"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">14%20PM</span><span class="informations">481×553 90 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>

ajbrock: <p>Are you using multithreading (particularly a multi-threaded dataloader)? Sometimes not all the child processes die with the main process and you need to check top or ps and kill them yourself.</p>

rasbt: <p>That’s a good point, I am indeed using a multi-threaded data loader! Hm, I can’t see any child processes hanging around though.</p>
<p>EDIT: I just checked again, the displayed memory use of the cards 1 &amp; 2 is indeed not just a nvidia-smi display issue, I can’t run anything on these cards until I clear out the memory manually</p>

Diego: <p>I have the same issue using a multi-threaded dataloader on a GTX 1060, sometimes it frees the memory after <code>cuda runtime error (2) : out of memory)</code> and sometimes it doesn’t which is a bit strange.</p>

SimonW: <p>Just to make sure that I understand correctly. You are saying that nvidia-smi shows the memory as held even if the Python process running PyTorch is terminated by the cuda OOM error, right?</p>

rasbt: <p>Yes that’s correct. And it’s not only nvidia-smi showing this (e.g., because of some caching issue) but the card really holds that memory (I can’t run any code on these cards anymore if they are not reset)</p>

SimonW: <p>I’ve seen similar things happen if the processes holding memory is<br>
something like a python subprocess. Could you check if your running<br>
processes don’t contain something that look suspicious?</p>

rasbt: <p>I actually quit all python processes but the memory is still not freed after that.</p>

SimonW: <p>How did you quit them? It’s not necessary consequence that subprocesses<br>
will terminate if main process quits.</p>

smth: <p><span class="mention">@rabst</span><br>
so, I remember this issue. When investigating, we found that there’s actually a bug in python multiprocessing that might keep the child process hanging around, as zombie processes.<br>
It is not even visible to <code>nvidia-smi</code>.</p>
<p>The solution is <code>killall python</code>, or to <code>ps -elf | grep python</code>  and find them and <code>kill -9 [pid]</code> to them.</p>

rasbt: <p>Thanks for asking <a class="mention" href="/u/simonw">@SimonW</a></p>
<p>I actually killed all main process via <code>ps</code>, but I forgot to call <code>ps -aux</code> which showed the child processes hanging around. Killing these helped freeing the memory. Thanks!</p>
<p><a class="mention" href="/u/smth">@smth</a></p>
<p>Solved my problem, thanks! Do you know if that’s a known bug listed on the Python bugtracker? Otherwise it might be worth adding it. Or maybe as a warning to users, it might be useful to modify the Runtime error message to warn users about that if possible?</p>

amitmore17: <p>I faced the same issue.<br>
I use following command to know the child processes causing the issue:</p>
<blockquote>
<blockquote>
<p>fuser -v /dev/nvidia*</p>
</blockquote>
</blockquote>
<p>Killing the listed processes solved the issue.</p>
<aside class="quote no-group" data-username="smth" data-post="14" data-topic="13775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>killall python</p>
</blockquote>
</aside>
<p>is “killall python” going to kill other python processes running on the system? Say some other python simulation running on system?</p>

ptrblck: <p>Yes, <code>killall NAME</code> will send a <code>SIGTERM</code> to all processes matching the name.</p>

jianchao-li: <aside class="quote no-group" data-username="smth" data-post="14" data-topic="13775">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/smth/48/13_2.png" class="avatar"> smth:</div>
<blockquote>
<p>It is not even visible to <code>nvidia-smi</code></p>
</blockquote>
</aside>
<p>Thank you so much for sharing the information. I actually came across this bug today when I terminated a PyTorch program which used a multi-threaded data loader. After killing the program, I tried to rerun it again but kept seeing CUDA OOM error. Running <code>nvidia-smi</code> showed everything was fine. And <code>cudaMemGetInfo</code> also did not expose the problem. For more details, please refer to <a href="https://github.com/pytorch/pytorch/issues/1829#issuecomment-430150771" rel="noopener nofollow ugc">this reply</a>.</p>
<p>Now I think I understand why it “worked” that way.</p>

ydukler: <p>When doing multi-gpu training with <code>DistributedDataParallel</code>, the zombie processes will not show up in nvidia-smi. But I found the following command helpful for killing zombie processes,</p>
<p><code>ps x|grep python|awk '\''{print $1;}'\'' |xargs kill -9</code></p>

Cherrybruin: <p>I have the same proble.<br>
first, I open a python shell, type <code>import torch</code><br>
this time I open another ssh type <code>watch nvidia-smi</code><br>
second I return to first python shell, create a tensor(27,3,480,270) and move it to cuda<br>
<code>input = torch.rand(27,3,480,270).cuda()</code><br>
the page of <code>nvidia-smi</code> change, and cuda memory increase<br>
third, use <code>ctrl+Z</code> to quit python shell.<br>
The cuda memory is not auto-free. The <code>nvidia-smi</code> page indicate the memory is still using.</p>
<p>The solution is you can use <code>kill -9 &lt;pid&gt;</code> to kill and free the cuda memory by hand.</p>
<p>I use Ubuntu 1604, python 3.5, pytorch 1.0.<br>
Although the problem solved, it`s uncomfortable that the cuda memory can not automatically free</p>