GPU memory is released only after error output in notebook

George_Fedoseev: <p>Hello!</p>
<p>I am doing training on GPU in Jupyter notebook.<br>
I have a problem: whenever I interrupt training GPU memory is not released. So I wrote a function to release memory every time before starting training:</p>
<pre><code class="lang-python">def torch_clear_gpu_mem():
    gc.collect()
    torch.cuda.empty_cache()
</code></pre>
<p>It releases some but not all memory: for example <strong>X</strong> out of 12 GB is still occupied by something. And it seems like this <strong>X</strong> is growing after every training interruption. And I can release it fully only by restarting the kernel.</p>
<p><strong>But I found a strange workaround: to cause some error to before calling <code>torch_clear_gpu_mem()</code></strong> - for example dividing by 0 in some cell of the notebook.<br>
Then after I call <code>torch_clear_gpu_mem()</code> memory is fully released!<br>
Can someone please explain how it happens? Is it some memory leak in Jupyter?<br>
I would like to make a function that does this automatically. Right now I need to call a special cell that causes error before clearing GPU memory.</p>

ptrblck: <p>If you are working interactively in your notebook, all objects including tensors will be stored.<br>
Even if you delete all tensors and clear the cache (usually not necessary), the CUDA context will still be initialized and will take some memory.</p>
<p>I guess the error is killing the kernel and thus the memory is completely released.</p>

George_Fedoseev: <p>Hello <a class="mention" href="/u/ptrblck">@ptrblck</a>! Thanks for answering.</p>
<p>Yes, I think CUDA context takes around 900MB for me, but after interruption I get 3-4GB that I cannot release (by deleting all objects / exiting function scope).<br>
I produce some simple exception (like ZeroDivision) and it doesn’t kill kernel (cause I can still access all my variables from other cells after error).<br>
But when I then call <code>torch_clear_gpu_mem()</code>, CUDA memory reliably returns to 900MB.</p>

ptrblck: <p>If you create the division by zero and raise the exception, all GPU memory is cleared and you can still access all CUDATensors?</p>

George_Fedoseev: <p>I can then start my training process on GPU again with all memory available (like after kernel restart)</p>

ptrblck: <p>OK, this would mean the exception is in fact restarting the kernel.<br>
What did you mean by “I can still access all my variables from other cells after error”?</p>
<p>If you still can access and e.g. print old variables, the kernel should be alive and it’s strange that all GPU memory is cleared.<br>
On the other hand, if you cannot print old variables and just restart the training, what would the difference be to a clean restart instead of a restart caused by an exception?</p>

George_Fedoseev: <p>I have a <strong>train_cell</strong>: where I call my training function (there I have a function and I call it right after definition).<br>
But this cell requires all prev cells (<strong>prep_cells</strong>) to be executed (imports, data preparation, etc.)</p>
<p>I also have an <strong>exception_cell</strong> where I do <code>1/0</code> (ZeroDivision exception).<br>
And <strong>mem_clear_cell</strong> where I call <code>torch_clear_gpu_mem()</code></p>
<ol>
<li>I run <strong>prep_cells</strong>, <strong>train_cell</strong>
</li>
<li>Interrupt training, GPU memory usage: 9GB</li>
<li>Run <strong>mem_clear_cell</strong>, GPU memory usage: 4GB (no matter how many times I run clear, gpu memory usage doesn’t go down)</li>
<li>Run <strong>exception_cell</strong>, then <strong>mem_clear_cell</strong>, GPU memory usage: 935MB</li>
<li>Then I can just call  <strong>train_cell</strong> without calling <strong>prep_cells</strong>
</li>
</ol>
<p>So with exception approach to clearing GPU memory I don’t need to restart kernel and run <strong>prep_cells</strong> before running <strong>train_cell</strong></p>