Moving from GPU to CPU not freeing GPURAM

username: JBCoding
cooked: <p>I’m not sure whether this is a bug or I’m not understanding how pytorch works, hence why I’m posted here first. I’m using pytorch 1.5.0.</p>
<p>My model is running on the gpu and I convert each batch to the device at the beginning, then forward through the model. When I then move it to CPU however, it doesn’t seem to free the GPU memory. When the loop comes around again, the memory still isn’t freed and ends up with an out of memory issue after a few loops.</p>
<pre><code>mem_a = torch.cuda.memory_summary(device, True)

b = b.to(device)

mem_a2 = torch.cuda.memory_summary(device, True)

forward = model.forward(b)

mem_a21 = torch.cuda.memory_summary(device, True)

forward = forward.to("cpu")

mem_a3 = torch.cuda.memory_summary(device, True)
</code></pre>
<p>I’ve checked in the debugger, and the forward after the <code>.to()</code> statement says it is on the cpu, however if you look at the GPU ram below from these positions…</p>
<pre><code class="lang-auto">Loop 1:
mem_a = 238MB
mem_a2 = 778MB
mem_a21 = 3471MB
mem_a3 = 3471MB

Loop 2:
mem_a = 3471MB
mem_a2 = 3781MB
mem_a21 = 5382MB
mem_a3 = 5382MB

Loop 3:
mem_a = 5382
mem_a2 = 5693
Out of Memory
</code></pre>
<p>To me, it very much looks like pytorch isn’t clearing up the link to the GPU after moving to cpu. I’ve tried this in a number of ways, even separating the gpu/cpu variable and deleting the gpu one still doesn’t work, as if it maintains a link somewhere. If I delete while it’s still on gpu the memory usage disappears, so it definitely seems to be something around this <code>.to()</code> statement that is keeping a link to the GPU.</p>
<p>I’m needing to store the output of this all before being able to do what I do next, and have more than enough RAM for it if I can get it put properly get it out of the GPU, however I can’t seem to do that.</p>
<p>Am I doing something wrong here?</p>

username: Krish
cooked: <p>Can you try this one out:</p>
<pre><code class="lang-auto">torch.cuda.empty_cache()
</code></pre>

username: JBCoding
cooked: <p>I’ve tried adding it as below, but there has been no difference. I also tried <code>gc.collect</code> just to see if there was a variable being held within memory with no difference.</p>
<pre><code class="lang-auto">mem_a21 = torch.cuda.memory_summary(device, True)

forward = forward.to("cpu")
torch.cuda.empty_cache()

mem_a3 = torch.cuda.memory_summary(device, True)
</code></pre>
<p>I’m really unsure of where to debug from this point forward… I’ve also tried <code>forward.cpu()</code> and <code>torch.device("cpu")</code> so I don’t think it’s specifically an issue with the <code>.to()</code>.</p>

username: thomasjo
cooked: <p>Are you collecting results, e.g. losses, and storing them in a list or something similar? If you are, make sure you store graph-detached copies of the relevant tensors by calling <a href="https://pytorch.org/docs/stable/autograd.html#torch.Tensor.detach" rel="nofollow noopener"><code>Tensor.detach</code></a>.</p>
<pre><code class="lang-python">losses = []

# Inside training loop or similar
loss = criterion(...)
losses.append(loss.detach())
</code></pre>
<p>If you store tensors attached to a graph, invoking garbage collection will have no impact since those tensors are still actively referenced via the list.</p>

username: Jianxiang_Feng
cooked: <p>Hi everyboday,<br>
any updates for this issue?<br>
It seems that I came into the same situation.</p>
<p>Best,<br>
Jianxiang</p>

username: JBCoding
cooked: <p>In my case, it turns out that the now marked answer (that I forgot to mark as solution back then) was the issue/solution in my case.</p>

username: Jianxiang_Feng
cooked: <p>Thank you for the quick reply!<br>
Since I got into this during inference in which no gradients are required, I found that <a href="https://stackoverflow.com/questions/55322434/how-to-clear-cuda-memory-in-pytorch" rel="noopener nofollow ugc">other people</a> suggested to use <code>with torch.no_grad():</code> before doing inference in the mini-batch. And it worked fine for me.</p>

username: Ilker_GURCAN
cooked: <p>detach() is the key <img src="https://discuss.pytorch.org/images/emoji/apple/slight_smile.png?v=12" title=":slight_smile:" class="emoji" alt=":slight_smile:" loading="lazy" width="20" height="20"> Thanks <a class="mention" href="/u/thomasjo">@thomasjo</a></p>