What is the best way of creating a queue in pytorch? Why does apending to a list cause Cuda to run out memory?

username: knoriy
cooked: <p>I’ve created a queue in my forward function as shown below:</p>
<pre><code class="lang-auto">que_a = [torch.rand((1,256,256)) ] * 3 # init queue with len of 3
que_b = [torch.rand((1,256,256)) ] * 3 # init queue with len of 3

i=0
while i&lt;1e7:
    # Model does something
    que_a.pop(0); que_b.pop(0) # we pop the first element in the list
    que_a.append(var1); que_b.append(var2) # var 1 &amp; 2 is pushed to the stack
    i+=1
</code></pre>
<p>Over time this caused Cuda to run out of memory, Why does this happen?</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 7.79 GiB total capacity; 4.50 GiB already allocated; 82.88 MiB free; 4.54 GiB reserved in total by PyTorch)
</code></pre>

username: ptrblck
cooked: <p>I cannot reproduce this issue using a simple CUDAtensor:</p>
<pre><code class="lang-python">l = [torch.randn(1, device='cuda')]

for i in range(100):
    print('iter{}, mem allocated {}MB'.format(i, torch.cuda.memory_allocated()/1024**2))
    l.pop()
    l.append(torch.randn(1, device='cuda'))
</code></pre>
<p>and will get the same memory usage.</p>
<p>However, based on your comment:</p>
<pre><code class="lang-python"># Model does something
</code></pre>
<p>I guess that <code>var1</code> and <code>var2</code> might be created by the model. If that’s the case, note that each tensor could still be attached to the computation graph, so that not only the tensor itself, but the entire graph would be stored in the lists, if you don’t <code>detach()</code> these tensors.</p>

username: knoriy
cooked: <p>Thank you very much <a class="mention" href="/u/ptrblck">@ptrblck</a>, you are correct that the model creates <code>Var1</code> and <code>Var2</code>. I’ve just tested your solution of detaching the tensor and it solved the issue.</p>
<p>I will write a detailed destription of the problem and the solution here shortly.</p>

username: knoriy
cooked: <p>Following my previous reply:</p>
<p>I’ve created this simplified model that replicates the issue:</p>
<pre><code class="lang-auto">import torch
import torch.nn as nn
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

class Model(nn.Module):
    def __init__(self) -&gt; None:
        super().__init__()
        self.que = [torch.rand((1,2560,2560), device=device)] * 3 # Init queue with len of 3
        self.fc = nn.Linear(2560, 2560)
        
    def forward(self, x):
        t=0
        while t&lt;=1000:
            print("Itter count: ", t)
            x = self.fc(x)#.detach() # uncomment .detach to fixes the CUDA out of memory
            self.que.pop(0); self.que.append(x) # pop first element from the que and append x to the end of que
            t+=1

        return x

if __name__ == '__main__':
    model = Model().to(device)
    x=torch.rand((1,2560,2560), device=device)

    model(x)
</code></pre>
<p>I’ve also noticed that assigning the output of <code>self.fc</code> to a different variable also fixed the issue of Cuda running out of memory:</p>
<pre><code class="lang-auto">def forward(self, x):
    t=0
    while t&lt;=1000:
        print("Itter count: ", t)
        y = self.fc(x)
        self.que.pop(0); self.que.append(y) # pop first element from the que and append y to the end of que
        t+=1

    return y
</code></pre>

username: ptrblck
cooked: <p>Thanks for the update!<br>
I’m not sure, if the <code>self.que</code> usage is really the culprit here and think that you are increasing the memory by reusing <code>x</code> in the first example.</p>
<pre><code class="lang-python">        while t&lt;=1000:
            x = self.fc(x)
</code></pre>
<p>This code snippet will pass the output as the new input to the model, so that intermediate activations cannot be freed. The storage in <code>self.que</code> might be irrelevant here. Even if you <code>append</code> and <code>pop</code> it properly, the model itself would still hold references to the “old” input and PyTorch won’t free the graph.</p>