Debugging gpu memory leaks might be tricky with debugger

username: stas
cooked: <p>While debugging a program with a memory leak I discovered that the leak was bigger when I was using pycharm debugger. I haven’t compared this to other debuggers but there was a definite much larger gpu memory consumption.</p>
<p>I tried a whole bunch of debugger settings, including “on Demand” but none seem to make a difference.</p>
<p>(note: This post has been edited to add this clarification - as I originally blatantly blamed the pycharm debugger, but as <a class="mention" href="/u/googlebot">@googlebot</a> pointed out in his comments it could be just as well the case with any other debugger. )</p>
<p>the original post follows:</p>
<hr>
<p>Is anybody using pycharm debugger with pytorch programs? it leaks gpu memory like there is no tomorrow. This sounds similar to the problem with the backtrace on OOM in ipython not freeing gpu memory.</p>
<p>Unknowingly I made the mistake of actually trying to debug memory leakage using pycharm, not realizing that pycharm debugger itself is storing tensors and not freeing them, even with forced <code>gc.collect()</code>. I suppose this is by design since pycharm stores all the variables for the user to access and thus they can’t be freed until the frames are exited.</p>
<p>I discovered that while writing a script to reproduce a memory leak, and only when I added a gpu memory tracing in it using I noticed that that I was getting totally different measurements when running the same script under debugger and not.</p>
<p>I tried a whole bunch of settings, but none seem to help. If you have a way to tell pycharm to not store intermediary vars please share, but somehow I doubt it’s even possible.</p>
<p>So if you’re trying to debug a pytorch program under pycharm and you end up getting OOM, this is why.</p>

username: googlebot
cooked: <ol>
<li>Use “Variable loading policy” = on demand.</li>
<li>IPython’s history may keep tensors alive (underscore variables like _, _10). In practice, I almost never have issues caused by that, but maybe your console usage patterns are different. So, disabling IPython may do something (I don’t know how to disable history only).</li>
</ol>
<p>If you don’t pause or use breakpoints, I don’t see how pycharm would allocate cuda memory.</p>

username: stas
cooked: <p>Thank you for your follow up, <a class="mention" href="/u/googlebot">@googlebot</a></p>
<blockquote>
<p>Use “Variable loading policy” = on demand.</p>
</blockquote>
<p>As I mentioned I tried many different options, including this one - to no avail.</p>
<blockquote>
<p>Python’s history may keep tensors alive (underscore variables like _, _10)</p>
</blockquote>
<p>No, this has to do with ipython not releasing GPU ram on OOM - a huge problem for jupyter users. A fix has been proposed almost 2 years ago, but it has never been integrated:</p><aside class="onebox githubpullrequest" data-onebox-src="https://github.com/ipython/ipython/pull/11572">
  <header class="source">

      <a href="https://github.com/ipython/ipython/pull/11572" target="_blank" rel="noopener nofollow ugc">github.com/ipython/ipython</a>
  </header>

  <article class="onebox-body">
    <div class="github-row">



    <div class="github-icon-container" title="Pull Request">
      <svg width="60" height="60" class="github-icon" viewBox="0 0 12 16" aria-hidden="true"><path fill-rule="evenodd" d="M11 11.28V5c-.03-.78-.34-1.47-.94-2.06C9.46 2.35 8.78 2.03 8 2H7V0L4 3l3 3V4h1c.27.02.48.11.69.31.21.2.3.42.31.69v6.28A1.993 1.993 0 0 0 10 15a1.993 1.993 0 0 0 1-3.72zm-1 2.92c-.66 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2zM4 3c0-1.11-.89-2-2-2a1.993 1.993 0 0 0-1 3.72v6.56A1.993 1.993 0 0 0 2 15a1.993 1.993 0 0 0 1-3.72V4.72c.59-.34 1-.98 1-1.72zm-.8 10c0 .66-.55 1.2-1.2 1.2-.65 0-1.2-.55-1.2-1.2 0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2zM2 4.2C1.34 4.2.8 3.65.8 3c0-.65.55-1.2 1.2-1.2.65 0 1.2.55 1.2 1.2 0 .65-.55 1.2-1.2 1.2z"></path></svg>
    </div>

  <div class="github-info-container">



      <h4>
        <a href="https://github.com/ipython/ipython/pull/11572" target="_blank" rel="noopener nofollow ugc">fix a memory leak on exception (caused by the stored traceback)</a>
      </h4>

    <div class="branches">
      <code>ipython:main</code> ← <code>stas00:leak-on-exc</code>
    </div>

      <div class="github-info">
        <div class="date">
          opened <span class="discourse-local-date" data-format="ll" data-date="2019-01-22" data-time="01:54:55" data-timezone="UTC">01:54AM - 22 Jan 19 UTC</span>
        </div>

        <div class="user">
          <a href="https://github.com/stas00" target="_blank" rel="noopener nofollow ugc">
            <img alt="stas00" src="https://avatars.githubusercontent.com/u/10676103?v=4" class="onebox-avatar-inline" width="20" height="20">
            stas00
          </a>
        </div>

        <div class="lines" title="1 commits changed 1 files with 2 additions and 0 deletions">
          <a href="https://github.com/ipython/ipython/pull/11572/files" target="_blank" rel="noopener nofollow ugc">
            <span class="added">+2</span>
            <span class="removed">-0</span>
          </a>
        </div>
      </div>
  </div>
</div>

  <div class="github-row">
    <p class="github-body-container">Currently ipython stores the traceback on exception. The problem is that it crea<span class="show-more-container"><a href="https://github.com/ipython/ipython/pull/11572" target="_blank" rel="noopener nofollow ugc" class="show-more">…</a></span><span class="excerpt hidden">tes a circular reference to locals() in each frame of the traceback. And until next exception arrives gc.collect() will fail to collect. If locals() happened to be huge variables, like 8GB of GPU RAM, the user has no choice but to restart ipython/jupyter to recover.

Solution - cleanse the saved tb from any references to `locals()` by running it through: `traceback.clear_frames(tb)`

My fix was made to only one such location in the ipython code and there is a dozen other places where it does that. So please kindly review for other similar situations.

I wrote a test case that demonstrates the problem and putting this fix in that particular location fixed that leak. But I don't know whether they are other situations where a similar fix needs to be applied.

You can see the test case here (run it with Alt-F so it runs all cells ignoring the errors)
https://hub.mybinder.org/user/stas00-fastai-misc-xbrsnjae/notebooks/debug/ipython/locals_leak_on_exc.ipynb
It shows how if there is a local variable that is of 128MB in size, an exception involving that function will leak that much RAM. And it shows how the next exception will release the previously held locals() allowing gc.collect() to recover that leaked memory.

The source of the test case is here:
https://github.com/stas00/fastai-misc/blob/master/debug/ipython/locals_leak_on_exc.ipynb

and here it is in straight ipython's output:

```
In [1]: import numpy as np                                                                                                                                                  

In [2]: def consume_cpu_ram(n): return np.ones((n, n)) 
   ...: def consume_cpu_ram_128mb():  return consume_cpu_ram(2**12)                                                                                                         

In [3]: import gc, os, sys, time, psutil 
   ...: process = psutil.Process(os.getpid()) 
   ...: def cpu_ram_used():  return process.memory_info().rss                                                                                                               

In [4]: def fail(): 
   ...:     x = consume_cpu_ram_128mb() 
   ...:     raise ValueError("Ouch") 
   ...:                                                                                                                                                                     

In [5]: before = cpu_ram_used() 
   ...: fail()                                                                                                                                                              
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-5-3b8b03759f74&gt; in &lt;module&gt;
      1 before = cpu_ram_used()
----&gt; 2 fail()

&lt;ipython-input-4-e846ddbe1c18&gt; in fail()
      1 def fail():
      2     x = consume_cpu_ram_128mb()
----&gt; 3     raise ValueError("Ouch")
      4 

ValueError: Ouch

In [6]: gc.collect() 
   ...: after = cpu_ram_used() 
   ...: print(f"Difference { (after-before)/2**20}")                                                                                                                        
Out[6]: 1105
Difference 128.62890625

In [7]: # force ipython to reset its %tb 
   ...: raise ValueError("Reset")                                                                                                                                           
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
&lt;ipython-input-7-31374e7d7384&gt; in &lt;module&gt;
      1 # force ipython to reset its %tb
----&gt; 2 raise ValueError("Reset")

ValueError: Reset

In [8]: gc.collect() 
   ...: after = cpu_ram_used() 
   ...: print(f"Difference { (after-before)/2**20}")                                                                                                                        
Out[8]: 418
Difference 0.77734375
```</span></p>
  </div>

  </article>

  <div class="onebox-metadata">
    
    
  </div>

  <div style="clear: both"></div>
</aside>
<p>
I use <code>1/0</code> cell-fix following the oom cell to work around it.</p>
<blockquote>
<p>If you don’t pause or use breakpoints, I don’t see how pycharm would allocate cuda memory.</p>
</blockquote>
<p>Right, basically you’re saying do not use pycharm debugger.</p>
<p>It’s not allocating cuda memory - it prevents variables from being freed and <code>gc.collect()ed</code> and thus memory from being freed.</p>

username: googlebot
cooked: <p>Your message sounded like the debugger is somehow defective, which is not the case in my experience. IPython/jupyter’s leaks would happen regardless of pycharm or other debugger IDE, won’t they?</p>

username: stas
cooked: <p>You’re correct that I made a broad statement, without comparing to other debuggers. I appreciate you flagging that, <a class="mention" href="/u/googlebot">@googlebot</a>. I edited the first post to reflect that.</p>
<p>While there is definitely a similarity I don’t think we can compare this to jupyter/ipython. In ipython you can control what variables remain in scope, whereas debuggers do their own tracking that you can’t control.</p>
<p>Unfortunately, I haven’t saved that particular code that was showing drastically different gpu memory usage  w/ and w/o debugger. I tried a few approaches now and the only correlation I found is the number of breakpoints in frames that had huge vars on cuda - more extra memory used when more breakpoints.</p>
<p>I may get a chance to investigate it more and if I do I will report back. But I’m surely going to be wary of watching memory usage under debugger and will have to check the usage patterns outside of debugger.</p>