CUDA out of memory How to fix?

Mr_Tajniak: <p>How to clear GPU cache<br>
torch.cuda.empty_cache() doesn’t work</p>

dejanbatanjac: <p>The first question I would ask is the number of GPU cores I have.<br>
For instance if I have 8 cores, then <code>torch.cuda.empty_cache()</code> should be rephrased as something like this:</p>
<pre><code class="lang-auto">how_many_gpus = torch.cuda.device_count()
    for _ in range(how_many_gpus):
        torch.cuda.set_device(_)
        torch.cuda.empty_cache()
</code></pre>
<p>What have you tried so far?</p>

Mr_Tajniak: <p>I have NVIDIA GeForce GTX 1060 6GB</p>

ptrblck: <p><code>.empty_cache</code> will only clear the cache, if no references are stored anymore to any of the data. If you don’t see any memory release after the call, you would have to delete some tensors before.</p>

Mr_Tajniak: <p>How can I clear cache? I am using this <a href="https://github.com/avinashpaliwal/Super-SloMo" rel="nofollow noopener">https://github.com/avinashpaliwal/Super-SloMo</a></p>

Mr_Tajniak: <p>I tried this and<br>
TypeError: device_count() takes 0 positional arguments but 1 was given</p>

Mr_Tajniak: <p>What is wrong with this</p>
<pre><code class="lang-auto">import torch
how_many_gpus = torch.cuda.device_count()
for _ in range(how_many_gpus):
        torch.cuda.set_device("cuda0")
        torch.cuda.empty_cache()
</code></pre>

dejanbatanjac: <aside class="quote no-group" data-username="Mr_Tajniak" data-post="3" data-topic="57046" data-full="true">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr_tajniak/48/16271_2.png" class="avatar"> Mr_Tajniak:</div>
<blockquote>
<p>I have NVIDIA GeForce GTX 1060 6GB</p>
</blockquote>
</aside>
<p>Have you meant to say you don’t have multiple GPUs? Can you confirm?</p>

dejanbatanjac: <aside class="quote no-group" data-username="Mr_Tajniak" data-post="7" data-topic="57046">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/mr_tajniak/48/16271_2.png" class="avatar"> Mr_Tajniak:</div>
<blockquote>
<p>What is wrong with this</p>
</blockquote>
</aside>
<p>Please check out the <a href="https://pytorch.org/docs/stable/notes/cuda.html" rel="noopener nofollow ugc">CUDA semantics document</a>.</p>
<p>Instead, <code>torch.cuda.set_device("cuda0")</code> I would use <code>torch.cuda.set_device("cuda:0")</code>, but in general the code you provided in your last update <a class="mention" href="/u/mr_tajniak">@Mr_Tajniak</a> would not work for the case of multiple GPUs.</p>
<p>In case you have a single GPU (the case I would assume) based on your hardware, what <a class="mention" href="/u/ptrblck">@ptrblck</a> said:</p>
<blockquote>
<p><code>.empty_cache</code> will only clear the cache, if no references are stored anymore to any of the data. If you don’t see any memory release after the call, you would have to delete some tensors before.</p>
</blockquote>
<p>This basically means PyTorch <code>torch.cuda.empty_cache()</code> would clear the PyTorch cache area inside the GPU.</p>
<p>You can check out the size of this area with this code:</p>
<pre><code class="lang-auto">import torch

import gc
def p():
    c = torch.cuda.memory_cached()
    print(f'cached   :{c}')
    a = torch.cuda.memory_allocated()
    print(f'allocated:{a}')
    f = torch.cuda.memory_cached()-torch.cuda.memory_allocated()
    print(f'free     :{f}')
    
torch.cuda.empty_cache()
p()
r = torch.randn(1, 128).cuda()
p()
</code></pre>
<p>Out:</p>
<pre><code class="lang-auto">cached   :0
allocated:0
free     :0
cached   :2097152
allocated:512
free     :2096640
</code></pre>
<p>See how PyTorch allocated 2Mb of cache just for storing this 128 floats. If you would <code>del r</code> followed by <code>p()</code> the GPU memory will be free again.</p>
<p>If you would have some objects you haven’t deleted make sure you delete them if they are not needed.</p>
<p>Why did PyTorch cached the memory in advance?<br>
To reuse it later. This is the idea of the cache. We assume this precious resource will be used later.</p>

Mr_Tajniak: <p>I have single GPU<br>
GeForce GTX 1060 6GB</p>

Mr_Tajniak: <p>to free the GPU cache I have to enter</p>
<pre><code class="lang-auto">import torch
p()
del r
</code></pre>
<p>Yes?</p>

dejanbatanjac: <p><a class="mention" href="/u/mr_tajniak">@Mr_Tajniak</a>, all I wanted to say you would need to deal with the variables that consume GPU memory smart.</p>
<p>I haven’t tested <a href="https://github.com/avinashpaliwal/Super-SloMo" rel="nofollow noopener">project</a> you mentioned, but I just saw you created the <a href="https://github.com/avinashpaliwal/Super-SloMo/issues/58" rel="nofollow noopener">issue</a>.</p>
<p>You may expect the feedback in there. Check if OOM is there if you use a smaller video format. Check what are the GPU memory requirements based on the video format you use (size of the frame).</p>

Mr_Tajniak: <p>ok thanks for help. In case of problems I will contact you, ok?</p>

dejanbatanjac: <blockquote>
<p>In case of problems I will contact you, ok?</p>
</blockquote>
<p>Sure. Why not.</p>