Deleting tensors in a list, class, or tuple does not delete the original tensor

username: xvdp
cooked: <p>Im rewriting my question because I have the answer.<br>
Problem: if you have declare cuda tensors in python and you pass them to functions using containers, and then need to delete them, you have to delete all references. see example below</p>
<p>or look at the gist<br>
<aside class="onebox githubgist">
  <header class="source">
      <a href="https://gist.github.com/xvdp/5ea32d4baff801c8d80960a8fa7b4594" target="_blank" rel="nofollow noopener">gist.github.com</a>
  </header>
  <article class="onebox-body">
    <h4><a href="https://gist.github.com/xvdp/5ea32d4baff801c8d80960a8fa7b4594" target="_blank" rel="nofollow noopener">https://gist.github.com/xvdp/5ea32d4baff801c8d80960a8fa7b4594</a></h4>
<h5>memory_tests.py</h5>
<pre><code class="Python">"""testing vram in pytorch cuda
every time a variable is put inside a container in python, to remove it completely
one needs to delete variable and container,
this can be problematic when using pytorch cuda if one doesnt clear all containers
Three tests:

&gt;&gt;&gt; python memory_tests list
    # creates 2 tensors puts them in a list, modifies them in place, deletes them
    # in place mod changes original tensors
    # list and both tensors need to be deleted</code></pre>
This file has been truncated. <a href="https://gist.github.com/xvdp/5ea32d4baff801c8d80960a8fa7b4594" target="_blank" rel="nofollow noopener">show original</a>

<p>
</p>

  </article>
  <div class="onebox-metadata">
    
    
  </div>
  <div style="clear: both"></div>
</aside>
</p>

username: xvdp
cooked: <p>I guess the answer is just have to be careful</p>
<p>if you declare</p>
<pre><code class="lang-python">t = torch.randn((1,3,1024,1024),device="cuda")
f = [t]
def something(f):
  f.mul_(0.001)
  return f
f1 = something(f)
</code></pre>
<p>to clear cuda memory you have to</p>
<pre><code class="lang-python">del t
del f
del f1
</code></pre>
<p>same goes for tuples, dictionaries. I even tried building my own tensorset class in cpp + pybind, still, as soon as it is used in python, passing it to a function and returning it creates a reference. To clear VRAM you have to delete all references.</p>
<p>If anyone has a better solution, please shout.<br>
Thanks</p>

username: fei.hu
cooked: <p>I agree with the above comments. It is so easy to miss some references. It has been more than one year since the original post. I’m wondering If there are any other convenient ways to delete tensors?</p>

username: xvdp
cooked: <p>Latest pytorch has much wider set of cuda management tools, but cleanup to my knowledge is not part of it ( im still on 1.5).</p>
<p>But one could go further than just cleaning up, There’s a lot of things that can be done in cuda much cleaner than in pytorch cuda, for instance common projection ops like,  <code>x.mm(x.T)</code> which cannot be done in place, (and if you require grad you couldnt anyway), but results of yet another tensor of the same size when in many occasions you only need half the upper triangle…</p>
<p>or some modification of the eye, that fills all the values,  such as <code>torch.eye(m).sub(1/m)</code>;<br>
that tensor is just 2 values <code>(1-1/m, -1/m)</code>; but if <code>m</code> happens to be large and you are using cuda, its a crapload of memory that you can be leaving around, they eye, the modified eye, and whatever you do with it. If you write it in a cuda loop it could be minimal informatio</p>
<p>Another one that I;d like automated in memory management, is one that I kind of premanage myelf but not fully - Lets say im testing batches of (256,3,256,256) on a bunch of networks, On a TitanRTX I can run those on Resnet18, 34 and 50, but not Rensnet101 or densnets or vgg -<br>
Itd be great to have a quick network eval flagging maximum number of bytes that it will be required to run the batch, or with a tiny bit more math, the batch sizes that it will accept.</p>
<p>These would be nice to have</p>