Clearing the GPU is a headache

alx: <p>Hi all, before adding my model to the gpu I added the following code:</p>
<pre><code class="lang-auto">def empty_cached():
  gc.collect()
  torch.cuda.empty_cache()
</code></pre>
<p>The idea buying that it will clear out to GPU of the previous model I was playing with.</p>
<p>Here’s a scenario, I start training with a resnet18 and after a few epochs I notice the results are not that good so I interrupt training, change the model, run the function above.<br>
When I do it that way I get a:</p>
<pre><code class="lang-auto">RuntimeError: CUDA out of memory. Tried to allocate 24.00 MiB (GPU 0; 11.17 GiB total capacity; 10.56 GiB already allocated; 9.81 MiB free; 10.85 GiB reserved in total by PyTorch)
</code></pre>
<p>However, if I interupt training, restart the kernel and run the same model that wouldn’t work before, it now works.</p>
<p>It seems like nothing works as good as restarting the kernel. What would come the closest to it?</p>

ptrblck: <p>You would have to delete all references to the tensors you would like to free.<br>
<code>gc.collect()</code> and <code>empty_cache()</code> will not free tensors, which are still used and referenced.</p>

alx: <p>I see!</p>
<p>How can I delete references to the tensors?</p>
<p>I tried with<code> del(model, optimizer, criterion)</code> but when I query the GPU memory before and after it seems like its the same.</p>

ptrblck: <p><code>del tensor</code> should work. To release the cached memory, you would need to call <code>torch.cuda.empty_cache()</code> afterwards.</p>
<p>Here is a small example:</p>
<pre><code class="lang-python">print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

x = torch.randn(1024*1024).cuda()

# 4MB allocation and potentially larger cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

y = torch.randn(8*1024*1024).cuda()

# 4+32=36MB allocation and potentially larger cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

del x

# 32MB allocation, cache should stay the same
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

torch.cuda.empty_cache()

# 32MB allocation and cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

del y

# 0MB allocation, 32MB cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)

torch.cuda.empty_cache()

# 0MB allocation and cache
print(torch.cuda.memory_allocated()/1024**2)
print(torch.cuda.memory_cached()/1024**2)
</code></pre>

alx: <p>Amazing, it works great!</p>
<p>I was using the <code>nvidia-smi</code> to determine the memory allocation on the GPU. What would be the difference between that and <code>torch.cuda.memory_allocated</code>.</p>

ptrblck: <p><code>nvidia-smi</code> shows the all processes which use memory on the device.<br>
So besides PyTorch or course other processes might use memory.<br>
Also, note that PyTorch loads the CUDA kernels, cudnn, CUDA runtime etc. after the first CUDA operation, which will also allocate memory (and cannot be freed until the script exits).<br>
Depending on the device, CUDA version etc. this CUDA context might take ~700MB.</p>

Saifeddine_Barkia: <p><a class="mention" href="/u/ptrblck">@ptrblck</a> sir, I have the same problem here, can you help me determine where I put the cuda. empty_cache()?</p><aside class="quote" data-post="4" data-topic="91385">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/shisho_sama/48/6926_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/runtimeerror-cuda-out-of-memory/91385/4">RuntimeError: CUDA out of memory?</a> <a class="badge-category__wrapper " href="/c/vision/5"><span data-category-id="5" data-drop-close="true" class="badge-category " title="Topics related to either pytorch/vision or vision research related topics"><span class="badge-category__name">vision</span></span></a>
  </div>
  <blockquote>
    Check the attention_maps and raw_features, as well.  If they are not scalers, and they do not participate in the training (contribute to it, and you are just using them as pure outputs) considert doing a detach() on them as well.
  </blockquote>
</aside>


ljj7975: <p>is there a way to programmatically clear CUDA context?</p>
<p>I would like to release the memory allocated by CUDA context after model training because my process need to do other operations after that</p>
<aside class="quote quote-modified" data-post="1" data-topic="126102">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ljj7975/48/12561_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/model-to-cpu-does-not-release-gpu-memory-allocated-by-registered-buffer/126102">Model.to("cpu") does not release GPU memory allocated by registered buffer</a> 
  </div>
  <blockquote>
    I have figured that registered_buffer does not release GPU memory when the model is moved back to CPU. 
Here is the minimal code for reproducing the observation 
import torch
from torch import nn
from subprocess import Popen, PIPE

class TestNet(nn.Module):
    def __init__(
        self
    ):
        super().__init__()
        self.register_buffer("test", torch.Tensor([1,2,3]).view(-1, 1, 1), False)

def log_nvidia_smi():
    print("--- nvidia-smi ---")
    for line in Popen('nvidia-smi', shel…
  </blockquote>
</aside>


ptrblck: <p>No, you cannot delete the CUDA context while the PyTorch process is still running and would have to shutdown the current process and use a new one for the downstream application.</p>

wiiiktor: <p>Is there a conveninent method for deleting all the model tensors in one go?</p>