Torch.cuda.empty_cache() and RuntimeError: CUDA out of memory

cbd: <p>I have custom loss function. If loss is positive i do backward() as mentioned below in code and after validation free the space using “torch.cuda.empty_cache()”</p>
<p>The code execute fine till the loss is positive. When loss is negative it doesn’t execute Training part mentioned in code as per logic but when it starts training for next batch it encounter “RuntimeError: Cuda out of memory”. After debugging i come to know when i <strong>coment</strong> <strong>torch.cuda.empty_cache()</strong> <strong>of validation part</strong>, error disappear. How <strong>comment on that line</strong> free up the space? Normally we use put that line to free up the space.</p>
<pre><code class="lang-auto">#### Training Part #####

if(loss&gt;0):
        self.backward_G()            
        self.optimizer_G.step() 
torch.cuda.empty_cache()
#### Validation Part #####

with torch.no_grad():
       # Validation image
torch.cuda.empty_cache()</code></pre>

ptrblck: <aside class="quote no-group" data-username="cbd" data-post="1" data-topic="147719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>Normally we use put that line to free up the space.</p>
</blockquote>
</aside>
<p>While it’s true that you are freeing the cache so that this memory can be used by other processes, you will not avoid OOM errors (internally <code>empty_cache()</code> will be called and the allocation will be re-executed), so calling <code>empty_cache()</code> manually will just slow down your code.</p>

cbd: <aside class="quote no-group" data-username="ptrblck" data-post="2" data-topic="147719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/48/1823_2.png" class="avatar"> ptrblck:</div>
<blockquote>
<p>internally <code>empty_cache()</code> will be called and the allocation will be re-executed</p>
</blockquote>
</aside>
<p>Ok.</p>
<aside class="quote no-group" data-username="cbd" data-post="1" data-topic="147719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<p>After debugging i come to know when i <strong>coment</strong> <strong>torch.cuda.empty_cache()</strong> <strong>of validation part</strong>, error disappear.</p>
</blockquote>
</aside>
<p>Why it gives this strange behaviour? and till loss is non negative it works fine.</p>

ptrblck: <p>I don’t know and it’s hard to tell how the memory is scattered. You could add debug print statements via <code>print(torch.cuda.memory_summary())</code> and check the allocations to see if anything stands out.</p>

cbd: <p>Ok. Thanks. i will look into it.</p>

cbd: <p>I just need to confirm that the below statement free some space when it execute or not. The reason is when loss is &gt;0, it runs normally. Just in case when it is negative, it doesn’t entered in this loop and throw memory error in next epoch.</p>
<aside class="quote no-group" data-username="cbd" data-post="1" data-topic="147719">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/c/13edae/48.png" class="avatar"> cbd:</div>
<blockquote>
<pre><code class="lang-auto">if(loss&gt;0):
        self.backward_G()            
        self.optimizer_G.step() 
</code></pre>
</blockquote>
</aside>

ptrblck: <p><code>self.backward_G()</code> will compute the gradients and free the intermediate forward activations, which would reduce the memory usage. I assume you are not calling <code>backward</code> if the loss is negative and might thus keep the computation graph (with the intermediate activations) alive, which could raise the OOM error in the next iteration.</p>