The gc mechanism doesn't show new tensors but graphics memory usage increases

username: 11130
cooked: <p>Hello, everyone. I use torch.cuda.max_memory_allocated() to track graphics memory usage of my program. I carefully delete tensors immediately after they are used but the graphic card still runs out of memory after several iterations.</p>
<p>Then I use the following snippet to track residing tensors.</p>
<pre><code class="lang-auto"># prints currently alive Tensors and Variables
import torch
import gc
for obj in gc.get_objects():
    try:
        if torch.is_tensor(obj) or (hasattr(obj, 'data') and torch.is_tensor(obj.data)):
            print(type(obj), obj.size())
    except:
        pass
</code></pre>
<p>After one SGD iteration, my program outputs</p>
<pre><code class="lang-auto">&lt;class 'torch.Tensor'&gt; torch.Size([100, 10])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([128, 784])
&lt;class 'torch.Tensor'&gt; torch.Size([128])
&lt;class 'torch.Tensor'&gt; torch.Size([64, 128])
&lt;class 'torch.Tensor'&gt; torch.Size([64])
&lt;class 'torch.Tensor'&gt; torch.Size([10, 64])
&lt;class 'torch.Tensor'&gt; torch.Size([10])
&lt;class 'torch.Tensor'&gt; torch.Size([100, 109386])
&lt;class 'torch.Tensor'&gt; torch.Size([300, 1, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([300])
&lt;class 'torch.Tensor'&gt; torch.Size([300, 109386])
&lt;class 'torch.Tensor'&gt; torch.Size([60000, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([60000])
&lt;class 'torch.Tensor'&gt; torch.Size([10000, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([10000])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([128, 784])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([128])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([64, 128])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([64])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10, 64])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10])
&lt;class 'torch.Tensor'&gt; torch.Size([50, 109386])
maximum graphics memory 2118.43798828125
</code></pre>
<p>The next SGD iteration shows exactly the same tensors:</p>
<pre><code class="lang-auto">&lt;class 'torch.Tensor'&gt; torch.Size([100, 10])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([])
&lt;class 'torch.Tensor'&gt; torch.Size([128, 784])
&lt;class 'torch.Tensor'&gt; torch.Size([128])
&lt;class 'torch.Tensor'&gt; torch.Size([64, 128])
&lt;class 'torch.Tensor'&gt; torch.Size([64])
&lt;class 'torch.Tensor'&gt; torch.Size([10, 64])
&lt;class 'torch.Tensor'&gt; torch.Size([10])
&lt;class 'torch.Tensor'&gt; torch.Size([100, 109386])
&lt;class 'torch.Tensor'&gt; torch.Size([300, 1, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([300])
&lt;class 'torch.Tensor'&gt; torch.Size([300, 109386])
&lt;class 'torch.Tensor'&gt; torch.Size([60000, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([60000])
&lt;class 'torch.Tensor'&gt; torch.Size([10000, 28, 28])
&lt;class 'torch.Tensor'&gt; torch.Size([10000])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([128, 784])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([128])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([64, 128])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([64])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10, 64])
&lt;class 'torch.nn.parameter.Parameter'&gt; torch.Size([10])
&lt;class 'torch.Tensor'&gt; torch.Size([50, 109386])
maximum graphics memory 2245.8056640625
</code></pre>
<p>But the memory goes up to 2245.8 (MB) from 2118.4 (MB). Why does this happen? Does python’s gc miss some tensors? Actually my program uses the operation below quite often, i.e., converting tensors to numpy arrays.</p>
<pre><code class="lang-auto">a=torch.norm(tensor_T, dim=1).detach().cpu().numpy()
</code></pre>
<p>Will this operation allow intermediate tensors to evade python’s garbage collection mechanism?</p>
<p>My program also has a lot of OrderedDicts, with keys being str’s and vals being tensors.</p>
<p>I tested my program on Titan X Pascal (with cuda 10.1, python 3.6.9 and pytorch 1.2.0) and Geforce RTX 2080 ti (with cuda 10.0, python 3.6.9 and pytorch 1.2.0).</p>
<p>Thank you for your attention!</p>

username: albanD
cooked: <blockquote>
<p>But the memory goes up to 2245.8 (MB) from 2118.4 (MB).</p>
</blockquote>
<p>This is a small increase. This might simply be because you have some slightly different behavior between the first iteration and the others (because the first iteration has only the results from that iteration, while during the second, some buffers from the 1st one might still be around).<br>
Does the memory continue to grow for the other iterations?</p>
<p>Also you’re looking at the maximum allocated memory, not the current one.</p>

username: 11130
cooked: <p>Yes, it grows after every SGD iteration and my graphics memory quickly runs out. I have converted my tracking codes to</p>
<pre><code class="lang-auto">def graphics_memory():
    xa = torch.cuda.memory_allocated()
    xa = xa / 1024 / 1024
    xb = torch.cuda.memory_cached()
    xb = xb / 1024 / 1024
    return max(xa, xb)

def memory_track():
    for obj in gc.get_objects():
        try:
            if torch.is_tensor(obj) or (hasattr(obj, "data") and torch.is_tensor(obj.data)):
                print(type(obj), obj.size())
            elif isinstance(obj, collections.OrderedDict):
                for key, val in obj.items():
                    if torch.is_tensor(val):
                        print("key {}, val {}".format(key, val.size()))
        except:
            pass
    print("current memory allocated {}".format(graphics_memory()))
    return
</code></pre>
<p>It still shows the current memory usage keeps growing despite that the residing tensors or OrderedDicts remain the same.</p>

username: albanD
cooked: <p>Could you give a small code sample (40-50 lines) that reproduces this behavior so that I can investigate locally please?</p>

username: 11130
cooked: <p>After an arduous day of debugging, I finally found the bug. Thanks a lot! Have a nice day!</p>

username: albanD
cooked: <p>Out of curiosity, what was it?</p>