CUDA memory is not freed properly

pmeier: <p>I think I’m missing something in my understanding of the CUDA memory management. I was under the impression that if one deletes all references to objects that were stored on a GPU and subsequently frees the cache, the allocated memory should be zero.</p>
<p>Since my code is part of a larger project and I was until now unable to reproduce the behaviour with a minimal example, I’ll show you a simplified version of what my code is doing. Consider the following snippet:</p>
<pre><code class="lang-python">import torch
from torchvision.models import vgg19

device = torch.device("cuda:0")

def display_memory():
    torch.cuda.empty_cache()
    memory = torch.cuda.memory_allocated(device)
    print("{:.3f} GB".format(memory / 1024 ** 3))

def fn():
    input = torch.rand(16, 3, 512, 512, device=device)
    model = vgg19().features.to(device)
    return model(input).detach()

display_memory()

output = fn()
print(output.size())
display_memory()

del output
display_memory()
</code></pre>
<p>This prints the following:</p>
<pre><code class="lang-auto">0.000 GB
0.009 GB
0.001 GB
</code></pre>
<p>As expected the <code>detach</code>ed output adds almost no extra memory. After the <code>del</code> statement the memory is almost completely freed. Since this could be related to my later question: Can someone explain the remaining 1 MB after all <code>Tensor</code>s and <code>nn.Modules</code> are <code>del</code>eted?</p>
<p>For my actual code the following is displayed if I insert the <code>display_memory()</code> function at the appropriate places:</p>
<pre><code class="lang-auto">0.000 GB
0.956 GB
0.946 GB
</code></pre>
<p>Even after <code>del</code>etion of every <code>Tensor</code> a large part of the memory remains occupied.</p>
<p>Can someone think of something what I could be doing wrong?</p>

ptrblck: <p>Usually the CUDA context uses some memory on your card which depends on the driver, GPU, CUDA version, etc. as far as I know.</p>

pmeier: <p>How much is</p>
<blockquote>
<p>some memory</p>
</blockquote>
<p>? That would explain the remaining 1 MB of the example, but I think almost 1 GB is too much, or am I wrong?</p>
<p>If that helps:</p>
<pre><code class="lang-auto">driver: 418.67
GPU:    GeForce GTX 1080
CUDA:   10.1
</code></pre>

ptrblck: <p>In my case, it seems to use approx <code>950MB</code> for:</p>
<pre><code class="lang-auto">driver: 418.56
GPU:    Titan V
CUDA:   10.1
</code></pre>

pmeier: <p>In that case I have some follow up questions:</p>
<ol>
<li>Do you have a resource where I can read about that?</li>
<li>Can this somehow be freed? In my case I need to call the <code>fn()</code> function multiple times and after a few iterations I don’t have enough memory left to execute it.</li>
</ol>

ptrblck: <ol>
<li>
<p>Based on <a href="https://devtalk.nvidia.com/default/topic/1044191/cuda-programming-and-performance/determine-memory-cuda-context-memory-usage/post/5304742/#5304742" rel="nofollow noopener">this topic</a> it seems the best way would be to directly measure the memory usage on your platform.</p>
</li>
<li>
<p>No, as the CUDA context holds the management data which is necessary to run CUDA applications.</p>
</li>
</ol>
<p>Are you seeing a growing memory usage, i.e. could you (accidentally) store the computation graph while this might not be necessary?</p>

pmeier: <p>I think you are right about the stored context information being the source of my problem. I’ve temporarily reduced the memory requirement of my <code>fn()</code> and tested it within a loop. After a few iterations the remaining memory stops to grow and stays at ~900 MB. I don’t think I accidentally store the computation graph, since</p>
<ol>
<li>I’ve <code>detach</code>ed the result of <code>fn()</code>, and</li>
<li>I’ve deleted every <code>Tensor</code> out of desperation without effect.</li>
</ol>
<hr>
<p>Do you have any idea why the size of CUDA context grows if I execute <code>fn()</code> multiple times? I was planning on doing something like this:</p>
<pre><code class="lang-python">def main():
    fn(parameter_set1)
    fn(parameter_set2)
    ...
</code></pre>
<p>If <code>fn()</code> needs more memory than what is available at maximum minus the CUDA context, the second call fails. Unfortunately this is the case for me.</p>