Unable to empty cuda cache

username: Andrea_Rosasco
cooked: <p>I’m trying to free some GPU memory so that other processes can use it. I tried to do that by executing <code>torch.cuda.empty_cache()</code> after deleting the tensor but for some reason it doesn’t seem to work.</p>
<p>I wrote this small script to replicate the problem</p>
<pre><code class="lang-auto">os.environ['CUDA_VISIBLE_DEVICES'] = '0'
showUtilization()
t = torch.zeros((1, 2**6, 2**6)).to(f'cuda')
showUtilization()
del t
torch.cuda.empty_cache()
showUtilization()
</code></pre>
<p>The memory utilization grows from 5% to 12% after allocating the tensor and stays to 12% even after emptying the cache.<br>
Of course as the process terminates the memory is released but I’d need to do that while the process is running. Does anyone have any idea about how to solve this?</p>

username: ptrblck
cooked: <p>Your approach should work as shown here:</p>
<pre><code class="lang-python">print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 0

t = torch.zeros((1, 2**6, 2**6)).to(f'cuda')
print(torch.cuda.memory_allocated())
&gt; 16384
print(torch.cuda.memory_reserved())
&gt; 2097152

del t
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 2097152

torch.cuda.empty_cache()
print(torch.cuda.memory_allocated())
&gt; 0
print(torch.cuda.memory_reserved())
&gt; 0
</code></pre>
<p>Note that the first CUDA operation will create the CUDA context on the device, which will load all kernels, cudnn, etc. onto the device.<br>
This memory is not reported by <code>torch.cuda.memory_allocated()</code> and <code>torch.cuda.memory_reserved()</code> and can be seen via <code>nvidia-smi</code>.</p>

username: Andrea_Rosasco
cooked: <p>Thanks for your response. If I run your code I get the exact same results but for some reason nvidia-smi doesn’t seem to notice that the memory was deallocated.<br>
If I run this code</p>
<pre><code class="lang-auto">def nvidia_smi():
    out = subprocess.Popen(['nvidia-smi'], stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    sout, serr = out.communicate()
    res = re.findall("([0-9]*)MiB / 16130MiB", str(sout))
    res = [int(x) for x in res]
    return res

os.environ['CUDA_VISIBLE_DEVICES'] = '2'
print(f'{nvidia_smi()[2]} MiB')

t = torch.zeros((1, 2**6, 2**6)).to(f'cuda')
print(f'{nvidia_smi()[2]} MiB')

del t
print(f'{nvidia_smi()[2]} MiB')

torch.cuda.empty_cache()
print(f'{nvidia_smi()[2]} MiB')
</code></pre>
<p>The result is</p>
<pre><code class="lang-auto">470 MiB
1473 MiB
1473 MiB
1471 MiB
</code></pre>
<p>As the process terminate the used memory goes down to 470 MiB again.<br>
For some <code>reason empty_cache()</code> manages to deallocate 2 MiB (this is consistent and not due to other processes on the same GPU I’ve tried it multiple times). Thinkig about it I guess that those 2 MiB are the size of the tensor I allocate. The other 1001 MiB are probably allocated by the CUDA backend for some internal functioning reason.</p>
<p>I was gonna ask if there’s a way to prevent that but I don’t think so.</p>

username: ptrblck
cooked: <aside class="quote no-group" data-username="Andrea_Rosasco" data-post="3" data-topic="99647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrea_rosasco/48/14939_2.png" class="avatar"> Andrea_Rosasco:</div>
<blockquote>
<p>For some <code>reason empty_cache()</code> manages to deallocate 2 MiB (this is consistent and not due to other processes on the same GPU I’ve tried it multiple times). Thinkig about it I guess that those 2 MiB are the size of the tensor I allocate.</p>
</blockquote>
</aside>
<p>Yes, the 2MB are shown in the <code>torch.cuda.memory_reserved()</code> output, which gives you the allocated and cached memory: <code>2097152 / 1024**2 = 2.0MB</code>.</p>
<aside class="quote no-group" data-username="Andrea_Rosasco" data-post="3" data-topic="99647">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/andrea_rosasco/48/14939_2.png" class="avatar"> Andrea_Rosasco:</div>
<blockquote>
<p>The other 1001 MiB are probably allocated by the CUDA backend for some internal functioning reason.</p>
<p>I was gonna ask if there’s a way to prevent that but I don’t think so.</p>
</blockquote>
</aside>
<p>Yes, these are used by the CUDA context as described before and cannot be freed.</p>