Memory is not released in Jupyter after KeyboardInterrupt

suvcss: <p>Every time I manually interrupt training, some memory remains stuck. For example, interrupting this tutorial - <a href="http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html" rel="nofollow noopener">http://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html</a> during training phase consumes about 500mb. Memory is not connected to any objects, deleting everything in the notebook’s scope doesn’t release memory. Interrupting training second time adds the same amount of leaked memory to the “pool”. Restarting kernel helps:slight_smile:<br>
Is there any way to release the memory or reset the graph similar to tf.reset_default_graph()?</p>

SpandanMadan: <p>Yes, I face the same problem with jupyter. Restarting kernel also does not do the trick. In fact, jupyter starts a bunch of processes with IDs in a sequence. They won’t even show up on your nvidia-smi command.</p>
<p>The only way i’ve found to find them is use</p>
<pre><code>sudo fuser -v /dev/nvidia*
</code></pre>
<p>Then, kill the processes using PID.</p>
<p>OR just kill all ipython kernel stuff using</p>
<pre><code>pkill -f ipykernel</code></pre>

suvcss: <p>I tested it on CPU, maybe because of this the restart did help</p>

suvcss: <p>Also repeatedly deep coping model to the same variable leads to memory consumption.</p>

SpandanMadan: <p>potentially deleting that might resolve the issue! Any way that you know to delete it?</p>

suvcss: <p>Noting specific to Pytorch:( Del and gc.collect() don’t help.</p>

tom: <p>I think Jupyter caches outputs in a way that adds references to the results.<br>
That said, I have been using Jupyter with pytorch for half a year now and have not run into the things you describe…</p>
<p>Best regards</p>
<p>Thomas</p>

suvcss: <p>Thomas, please, could you check it on your machine? It will take only a few minutes:) Run this tutorialhttp://pytorch.org/tutorials/beginner/transfer_learning_tutorial.html<br>
and interrupt training several times after maybe 10 seconds. Will your memory consumption grow?</p>

suvcss: <p>Tried few other things. The problem appears to be connected with how the Jupyter works. Would appreciate any ideas:)</p>

QuantScientist: <p>Have you tried:<br>
<code>%whos</code><br>
and then:<br>
<code>%reset -f</code><br>
and finally:<br>
<code>%whos</code></p>

SpandanMadan: <p>I tried this. This doesn’t release the GPU memory.</p>
<p>But this works -</p>
<pre><code>pkill -f ipykernel
</code></pre>
<p>This will kill your kernel too, so it will be like restarting the notebook.</p>

suvcss: <p>reset -f  doesn’t help to release memory, just clear every variable:)</p>

chenchr: <p>In the main page of the jupyter notebook you open, you can choose the ipynb you’ve just run, and on the top click shutdown, then I think the memory should be released.</p>

suvcss: <p>I know that I can shutdown or restart notebook to release memory:) The question is - how it can be done without restarting or why it happens?</p>

tom: <p>Have you tested whethet the same happens if you run one full pass before the partial ones?<br>
By design, pytorch caches allocations.</p>
<p>Best regards</p>
<p>Thomas</p>

QuantScientist: <p>I now have the exact same issue:</p>
<p><code>cuda runtime error (2) : out of memory at /home/gpu/dev/rt/pytorch/torch/lib/THC/generic/THCStorage.cu:66</code></p>
<p>Thrown after repeated training during which I stopped the kernel and started training again.<br>
Of-course PyTorch returns False for <code>torch.cuda.is_available()</code><br>
Neither torch nor Jupyter are able to recover from this. The only solution for me was to restart the computer since the memory is not released otherwise.</p>
<p>BTW I am using this command line for monitoring:<br>
<code>watch -n 0.1 'ps f -o user,pgrp,pid,pcpu,pmem,start,time,command -p</code>sudo lsof -n -w -t /dev/nvidia*<code>'</code><br>
(source: <a href="https://stackoverflow.com/questions/8223811/top-command-for-gpus-using-cuda" rel="nofollow noopener">https://stackoverflow.com/questions/8223811/top-command-for-gpus-using-cuda</a>)</p>

Sia_Rezaei: <p>I’m having the same issue. I’m developing a new model and naturally I get a lot of crashes as I’m debugging. Every time of have to restart the kernel, run the pre-processing steps, … It’s a pain in the neck.</p>