Memory leak persisting after closing python?

sad_robot: <p>I just had a memory leak that persisted with 90% of the GPU memory occupied AFTER I closed python and did <code>killall python</code> + <code>killall jupyter</code> for good measure. I also checked nvidia-smi and saw the 90% GPU memory usage but no corresponding processes actually using it…</p>
<p>How are things like this possible? How do I prevent them? And how do I diagnose them?<br>
The only recourse I had that I knew of at the time was to reboot. But I really need some context to this problem…</p>
<p>P.S. I just got this to happen again by interrupting a jupyter notebook cell during the training process. So that may be related. But still I’m not sure how to deal with these issues in the long run.</p>

ptrblck: <aside class="quote no-group" data-username="sad_robot" data-post="1" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>P.S. I just got this to happen again by interrupting a jupyter notebook cell during the training process.</p>
</blockquote>
</aside>
<p>Interrupting a process is often the root cause of a failed exit behavior. In my experience I haven’t seen dead processes often when working directly in the terminal (unless the code crashes) and have seen it more often in Jupyter (but I’m also not a heavy Jupyter user so others might have a different experience).</p>

sad_robot: <p>I see so it’s not a given in general that a process needs to be open to cause a memory leak?<br>
Is there some way to deal with these after they’ve occurred aside from rebooting?</p>
<p>Also out of curiosity what is your preference over Jupyter?</p>

ptrblck: <aside class="quote no-group" data-username="sad_robot" data-post="3" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>I see so it’s not a given in general that a process needs to be open to cause a memory leak?</p>
</blockquote>
</aside>
<p>I wouldn’t claim it’s a memory leak, but think it’s a dead process which still holds the memory.</p>
<aside class="quote no-group" data-username="sad_robot" data-post="3" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>Is there some way to deal with these after they’ve occurred aside from rebooting?</p>
</blockquote>
</aside>
<p>Finding the dead process via <code>ps aux</code> and killing it.</p>
<aside class="quote no-group" data-username="sad_robot" data-post="3" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>Also out of curiosity what is your preference over Jupyter?</p>
</blockquote>
</aside>
<p>I’m not a researcher and am more focused on the backend as well as debugging, so I wouldn’t necessarily stick to my setup, but:</p>
<ul>
<li>plain <code>vi</code> for remote work</li>
<li>Spyder for quick “frontend” work (coming from the signal processing domain with MATLAB, Spyder felt familiar but I think it’s not hugely popular)</li>
<li>VS Code (+ remote access if needed)</li>
</ul>

sad_robot: <aside class="quote no-group" data-username="ptrblck" data-post="4" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>killing</p>
</blockquote>
</aside>
<p>Thanks for the reply! Interesting choice in IDEs, I’ve been leaning that way too.<br>
But regarding killing the process I already tried killall python &amp; killall jupyter do you think it is some other process then?</p>

ptrblck: <p>I’m not sure what the exact naming would be, so I would recommend to check all processes in case it’s <code>jupyter-kernel</code> or so.</p>

sad_robot: <p>Thanks I’ll try that next time!</p>

gphilip: <aside class="quote no-group" data-username="sad_robot" data-post="5" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sad_robot/48/25510_2.png" class="avatar"> sad_robot:</div>
<blockquote>
<p>I already tried killall python &amp; killall jupyter</p>
</blockquote>
</aside>
<p>Try passing the <code>-9</code> flag to <code>killall</code>, so that it sends the stronger <code>SIGKILL</code> which cannot be blocked. E.g. <code>killall -9 jupyter</code>.</p>
<p>Killing <em>all</em> python processes may not be advisable, since you may end up disrupting non-ML programs. The better idea is to find the PIDs of your ML python programs using <code>ps -u $USER</code> and killing only the right ones using <code>kill -9 &lt;PID of the python process that you want to kill&gt;</code>.</p>

sad_robot: <aside class="quote no-group" data-username="gphilip" data-post="8" data-topic="131616">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/g/858c86/48.png" class="avatar"> gphilip:</div>
<blockquote>
<p>using</p>
</blockquote>
</aside>
<p>Idc, if I’m allowed to kill processes which should not be killed then that’s the Admin’s fault. Kill all is simpler, and prevents the possibility of unseen zombie processes.</p>
<p>Thanks for the tip about -9 flag though, its weird since it’s called killall I thought it would do SIGKILL by default? Good to know.</p>