Fully Clear GPU Memory after Evaluation

wwaayyaaww: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,  I am currently having the GPU memory leakage problem (<strong>during evaluation</strong>) that<br>
(1)   the GPU memory usage increased during evaluation, and<br>
(2)   it is not fully cleared after all variables have been deleted, and i have also cleared the memory using <code>torch.cuda.empty_cache()</code> and <code>gc.collect()</code>.</p>
<p>Do you have any idea on why the GPU remains occupied after the <strong>evaluation stage</strong>? and this problem occurs even if I set the batch size to 1.</p>
<p>My evaluation function is as follows:</p>
<pre><code class="lang-auto"> with torch.no_grad():
        
        for batch_idx, (x, y) in enumerate(loader_gallery):

            x = x.to(device)

            x, xx = model(x)
            xx = submodel(xx)

            gallery_fea_1 = torch.cat((gallery_fea_1, x.cpu()), 0)
            gallery_fea_2 = torch.cat((gallery_fea_2, xx.cpu()), 0)
            gallery_label = torch.cat((gallery_label, y.cpu()), 0)

            del x, xx, y
            time.sleep(0.0001)
</code></pre>
<p><div class="lightbox-wrapper"><a class="lightbox" href="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74cab48add5c447c02554a4be6539d1f3ff34663.png" data-download-href="https://discuss.pytorch.org/uploads/default/74cab48add5c447c02554a4be6539d1f3ff34663" title="gpu"><img src="https://discuss.pytorch.org/uploads/default/original/3X/7/4/74cab48add5c447c02554a4be6539d1f3ff34663.png" alt="gpu" data-base62-sha1="gFbGc37Ny6f54Zr0J4eAktwR0DV" width="690" height="193" data-dominant-color="F4F8FA"><div class="meta"><svg class="fa d-icon d-icon-far-image svg-icon" aria-hidden="true"><use href="#far-image"></use></svg><span class="filename">gpu</span><span class="informations">1081×303 8.15 KB</span><svg class="fa d-icon d-icon-discourse-expand svg-icon" aria-hidden="true"><use href="#discourse-expand"></use></svg></div></a></div></p>
<p>Thank you.</p>
<p>Sincerely.</p>

ptrblck: <p>Could you describe the “memory leak” a bit more?<br>
Are you seeing an increase in memory usage in the Windows task manager after executing the training and validation loop, which is approx. constant afterwards or is the memory increasing in each iteration until you run out of memory?<br>
The former could be expected, if you are keeping objects alive from the training loop and if the validation loop needs to allocate additional memory.<br>
PyTorch uses a custom caching allocator, which will keep the already allocated memory in the cache and will not release it for performance reasons.<br>
Calling <code>empty_cache</code> won’t avoid any out of memory issues, will slow down your code, and is useful if you want to release the cache so that other <em>processes</em> could use it.</p>

wwaayyaaww: <p>Hi <a class="mention" href="/u/ptrblck">@ptrblck</a>,</p>
<p>Thank you for your quick responses.</p>
<p>I encountered this problem in the evaluation mode (prior to training), and the GPU memory remains “constantly occupied” even after the evaluation has been completed.</p>
<p>I am finding ways to release this “occupied” GPU memory so that the training mode is executed without any GPU memory shortage issue.</p>
<p>And this problem only occurs if the pre-trained ResNet50 is used. Is that bcoz of the model complexity where a large portion of the GPU memory is allocated and kept in the cache even in the evaluation mode?</p>

ptrblck: <p>Unfortunately, I still don’t understand the issue correctly. Are you running out of memory during training or are you concerned about the memory cache (which will reuse memory as described before)?</p>

wwaayyaaww: <p>I apologize for the umbiguities. In fact, I am having problems during both training and testing.</p>
<p>(1)  Due to the reason that I am using a pre-learned modal, my practice is that I first ran in the evaluation mode (with some probing data) before training started to check for the initial performance.</p>
<p>(2)  I found that the GPU memory allocated for evaluation couldn’t be cleared (about 4G memory being occupied as illustated in the figure). This causes the memory shortage issue during training even if I set the batch size to only 16.</p>
<p>(3)  I am wondering if the the pre-allocated memory for the evaluation mode can be cleared for training purposes?</p>

ptrblck: <aside class="quote no-group" data-username="wwaayyaaww" data-post="5" data-topic="131219">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwaayyaaww/48/22536_2.png" class="avatar"> wwaayyaaww:</div>
<blockquote>
<p>I found that the GPU memory allocated for evaluation couldn’t be cleared (about 4G memory being occupied as illustated in the figure).</p>
</blockquote>
</aside>
<p>This is expected, since the memory would be in the <em>cache</em> assuming that you have deleted all tensors. If not delete objects, which are not needed anymore.</p>
<aside class="quote no-group" data-username="wwaayyaaww" data-post="5" data-topic="131219">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/wwaayyaaww/48/22536_2.png" class="avatar"> wwaayyaaww:</div>
<blockquote>
<p>This causes the memory shortage issue during training even if I set the batch size to only 16.</p>
</blockquote>
</aside>
<p>I assume that you’ve verified this claim by running the training without a previous evaluation and this setup worked fine?</p>