How to check memory leak in a model

username: VictorNi
cooked: <p>Hi all,</p>
<p>I implemented a model in PyTorch 0.4.0, but find that GPU memory increases at some iterations randomly. For example, in the first 1000 iterations, it uses GPU Mem 6G, and at a random iteration, it uses GPU Mem 10G.</p>
<p>I <code>del loss, image, label</code> and use <code>total loss += loss.item()</code> at each iteration, and conjecture that the model leaks memory sometimes.</p>
<p>I also tried to use gc print alive Tensors according to <a href="https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3?u=victorni">https://discuss.pytorch.org/t/how-to-debug-causes-of-gpu-memory-leaks/6741/3?u=victorni</a>, and found that there was a little difference between two iterations, but how could I figure out the reason?</p>
<p>Our model is <a href="https://github.com/twni2016/OrganSegRSTN_PyTorch/blob/master/OrganSegRSTN/model.py" rel="nofollow noopener">https://github.com/twni2016/OrganSegRSTN_PyTorch/blob/master/OrganSegRSTN/model.py</a>, in <code>forward()</code> we use random crop and other operations, maybe that caused memory leak?</p>
<p>Thank you!</p>

username: InnovArul
cooked: <p>A helpful comment from <a class="mention" href="/u/alband">@albanD</a>:</p>
<aside class="quote" data-post="2" data-topic="14530">
  <div class="title">
    <div class="quote-controls"></div>
    <img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/alband/48/215_2.png" class="avatar">
    <a href="https://discuss.pytorch.org/t/how-can-we-release-gpu-memory-cache/14530/2">How can we release GPU memory cache?</a> 
  </div>
  <blockquote>
    Hi, 
torch.cuda.empty_cache() (EDITED: fixed function name) will release all the GPU memory cache that can be freed. 
If after calling it, you still have some memory that is used, that means that you have a python variable (either torch Tensor or torch Variable) that reference it, and so it cannot be safely released as you can still access it. 
You should make sure that you are not holding onto some objects in your code that just grow bigger and bigger with each loop in your search.
  </blockquote>
</aside>


username: VictorNi
cooked: <p>Thanks for your reply.<br>
I have tried <code>torch.cuda.empty_cache()</code> after I <code>del model</code>, it really works that GPU memory was reduced to almost zero.<br>
However, my problem is that during the iterations of one model, GPU memory may increase. So should I insert <code>torch.cuda.empty_cache()</code> in every iteration?</p>

username: InnovArul
cooked: <p>I’m not sure if there is any efficient way than calling it in every iteration. I have noticed that <code>empty_cache()</code> slows down the process a bit,  so you have to compromise with that</p>

username: addisonklinke
cooked: <p><a class="mention" href="/u/victorni">@VictorNi</a> I had a leak in my training loop from tracking the total loss, and was able to resolve it by following the recommendations in this PyTorch FAQ <a href="https://pytorch.org/docs/stable/notes/faq.html#my-model-reports-cuda-runtime-error-2-out-of-memory" rel="nofollow noopener">article</a></p>